2023-12-16T21:01:25.130 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-16T21:01:25.130 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833 --owner scheduled_yuriw@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} --name yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/orig.config.yaml 2023-12-16T21:01:25.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T21:01:25.260 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833 branch: wip-yuri8-testing-2023-12-15-0911 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} email: yweinste@redhat.com first_in_suite: false job_id: '7494833' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-15-0911 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-15-0911 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2960 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2023-12-15-0911 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1a5SboxJoSrpG9mBywr6CCPu5P8Z3x4NaoXLuZ0bf4KeJcCsXygvslEt6eyjzJZfTB9xRERW+QgLRGV0099aw= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbkN8s8lHKeTZqUwEuj1Kn16eqSfxnbUrx/iZfNDoz3x1UwK7+tQOL2YiAs2mO3QT6yiHfQCZ6uKmanhTc6OTk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-16_15:45:14 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-16T21:01:25.260 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa; will attempt to use it 2023-12-16T21:01:25.261 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks 2023-12-16T21:01:25.261 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-16T21:01:25.262 INFO:teuthology.task.internal:Checking packages... 2023-12-16T21:01:25.288 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7586e95519b7be76f3b6e3a4b333888baf46bc9' 2023-12-16T21:01:25.288 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:01:25.288 INFO:teuthology.packaging:ref: None 2023-12-16T21:01:25.288 INFO:teuthology.packaging:tag: None 2023-12-16T21:01:25.288 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:01:25.288 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:01:25.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:01:25.506 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-202.gc7586e95 2023-12-16T21:01:25.507 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-16T21:01:25.514 INFO:teuthology.task.internal:no buildpackages task found 2023-12-16T21:01:25.515 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-16T21:01:25.550 INFO:teuthology.task.internal:Saving configuration 2023-12-16T21:01:25.567 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-16T21:01:25.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:01:25.599 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:56:13.427323', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1a5SboxJoSrpG9mBywr6CCPu5P8Z3x4NaoXLuZ0bf4KeJcCsXygvslEt6eyjzJZfTB9xRERW+QgLRGV0099aw='} 2023-12-16T21:01:25.625 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:56:13.425727', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbkN8s8lHKeTZqUwEuj1Kn16eqSfxnbUrx/iZfNDoz3x1UwK7+tQOL2YiAs2mO3QT6yiHfQCZ6uKmanhTc6OTk='} 2023-12-16T21:01:25.625 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-16T21:01:25.631 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-12-16T21:01:25.631 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-12-16T21:01:25.631 INFO:teuthology.run_tasks:Running task console_log... 2023-12-16T21:01:25.723 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f084470ae50>, signals=[15]) 2023-12-16T21:01:25.724 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-16T21:01:25.730 INFO:teuthology.task.internal:Opening connections... 2023-12-16T21:01:25.731 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:01:25.733 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:01:25.832 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:01:25.833 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:01:25.916 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-16T21:01:25.923 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2023-12-16T21:01:25.942 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2023-12-16T21:01:25.942 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2023-12-16T21:01:26.000 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2023-12-16T21:01:26.000 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T21:01:26.001 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T21:01:26.002 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2023-12-16T21:01:26.022 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-12-16T21:01:26.040 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-12-16T21:01:26.040 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-12-16T21:01:26.097 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2023-12-16T21:01:26.097 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2023-12-16T21:01:26.097 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2023-12-16T21:01:26.097 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T21:01:26.098 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T21:01:26.098 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-12-16T21:01:26.122 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-16T21:01:26.131 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-16T21:01:26.148 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-16T21:01:26.149 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2023-12-16T21:01:26.151 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2023-12-16T21:01:26.167 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-16T21:01:26.174 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-16T21:01:26.174 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2023-12-16T21:01:26.208 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2023-12-16T21:01:26.475 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-16T21:01:26.530 INFO:teuthology.run_tasks:Running task kernel... 2023-12-16T21:01:26.545 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-16T21:01:26.545 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-16T21:01:26.546 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T21:01:26.546 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T21:01:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:01:26.563 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-12-16T21:01:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:01:26.564 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-12-16T21:01:26.620 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-361.el9.x86_64 2023-12-16T21:01:26.620 INFO:teuthology.task.kernel:Running kernel on smithi121: 5.14.0-361.el9.x86_64 2023-12-16T21:01:26.620 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2023-12-16T21:01:26.622 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-361.el9.x86_64 2023-12-16T21:01:26.622 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-361.el9.x86_64 2023-12-16T21:01:26.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2023-12-16T21:01:28.774 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-12-16T21:01:28.797 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-12-16T21:01:30.223 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-12-16T21:01:30.416 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2023-12-16T21:01:31.232 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-12-16T21:01:32.280 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 7.9 MB 00:01 2023-12-16T21:01:36.760 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 18 MB 00:04 2023-12-16T21:01:41.190 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 5.8 MB 00:01 2023-12-16T21:01:42.524 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 15 kB 00:00 2023-12-16T21:01:44.170 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-12-16T21:01:45.180 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 18 MB 00:11 2023-12-16T21:01:48.792 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2023-12-16T21:01:50.948 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T21:01:50.949 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T21:01:51.045 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-12-16T21:01:51.050 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-12-16T21:01:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2023-12-16T21:01:51.050 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-12-16T21:01:51.050 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-12-16T21:01:51.050 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout:Install 4 Packages 2023-12-16T21:01:51.051 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:01:51.052 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 96 M 2023-12-16T21:01:51.052 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 121 M 2023-12-16T21:01:51.052 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-12-16T21:01:51.233 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 5.8 MB 00:02 2023-12-16T21:01:51.485 INFO:teuthology.orchestra.run.smithi121.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 17 MB/s | 5.6 MB 00:00 2023-12-16T21:01:51.802 INFO:teuthology.orchestra.run.smithi121.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 32 MB/s | 20 MB 00:00 2023-12-16T21:01:52.263 INFO:teuthology.orchestra.run.smithi121.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 34 MB/s | 38 MB 00:01 2023-12-16T21:01:52.594 INFO:teuthology.orchestra.run.smithi121.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 29 MB/s | 32 MB 00:01 2023-12-16T21:01:52.594 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:01:52.594 INFO:teuthology.orchestra.run.smithi121.stdout:Total 62 MB/s | 96 MB 00:01 2023-12-16T21:01:52.806 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2023-12-16T21:01:53.455 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-12-16T21:01:53.832 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-12-16T21:01:53.832 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-12-16T21:01:55.118 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-12-16T21:01:55.118 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-12-16T21:01:56.723 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 20 MB 00:03 2023-12-16T21:01:57.746 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-12-16T21:01:58.370 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:01:58.511 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:02:01.278 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:02:01.462 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:02:01.690 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2023-12-16T21:02:03.931 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T21:02:03.931 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T21:02:04.030 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-12-16T21:02:04.034 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:02:04.035 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2023-12-16T21:02:04.036 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2023-12-16T21:02:04.036 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2023-12-16T21:02:04.036 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:02:04.036 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 96 M 2023-12-16T21:02:04.036 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 121 M 2023-12-16T21:02:04.036 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2023-12-16T21:02:05.158 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 5.9 MB/s | 5.6 MB 00:00 2023-12-16T21:02:05.767 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 13 MB/s | 20 MB 00:01 2023-12-16T21:02:06.071 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:02:06.189 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:06.559 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 16 MB/s | 38 MB 00:02 2023-12-16T21:02:06.859 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 19 MB/s | 32 MB 00:01 2023-12-16T21:02:06.859 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:02:06.859 INFO:teuthology.orchestra.run.smithi046.stdout:Total 34 MB/s | 96 MB 00:02 2023-12-16T21:02:07.751 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2023-12-16T21:02:08.180 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2023-12-16T21:02:08.180 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2023-12-16T21:02:09.528 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2023-12-16T21:02:09.528 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2023-12-16T21:02:10.841 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2023-12-16T21:02:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:02:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:02:15.997 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:02:16.176 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:02:21.130 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:02:21.243 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:26.229 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:39.643 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:39.686 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:40.171 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:02:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:02:40.172 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:02:40.555 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-12-16T21:02:41.047 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:02:41.192 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:02:41.193 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:02:41.193 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T21:02:41.193 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T21:02:41.193 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T21:02:41.193 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-16T21:02:41.193 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T21:02:41.194 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:02:41.311 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:02:41.311 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:02:41.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2023-12-16T21:02:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2023-12-16T21:02:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2023-12-16T21:02:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2023-12-16T21:02:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2023-12-16T21:02:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2023-12-16T21:02:41.348 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T21:02:41.349 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T21:02:41.350 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T21:02:41.350 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2023-12-16T21:02:41.350 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:02:41.350 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T21:02:41.350 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T21:02:41.351 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 5.14.0-391.el9.x86_64 2023-12-16T21:02:41.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T21:02:41.959 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2023-12-16T21:02:43.300 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T21:02:43.329 INFO:teuthology.orchestra.run.smithi121.stderr:done 2023-12-16T21:02:43.331 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-12-16T21:02:43.346 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.ebYdY0BwfJ 2023-12-16T21:02:43.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ebYdY0BwfJ 2023-12-16T21:02:43.415 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.ebYdY0BwfJ 2023-12-16T21:02:43.555 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.ebYdY0BwfJ is 6KB 2023-12-16T21:02:43.604 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.ebYdY0BwfJ 2023-12-16T21:02:43.618 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T21:02:43.683 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T21:02:43.683 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T21:02:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T21:02:43.684 INFO:teuthology.orchestra.run.smithi121.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T21:02:43.685 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T21:02:43.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2023-12-16T21:02:56.832 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:56.877 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:57.255 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:57.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:02:57.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:02:57.255 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:02:57.697 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:02:57.697 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:02:57.697 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2023-12-16T21:02:57.698 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T21:02:57.698 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T21:02:57.698 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T21:02:57.698 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T21:02:57.698 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:02:57.698 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-12-16T21:02:58.157 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:02:58.302 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:02:58.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:02:58.302 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T21:02:58.302 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T21:02:58.303 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T21:02:58.303 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-16T21:02:58.303 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T21:02:58.303 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:02:58.426 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:02:58.426 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:02:58.426 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2023-12-16T21:02:58.479 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2023-12-16T21:02:58.480 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T21:02:58.481 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T21:02:58.481 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T21:02:58.481 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T21:02:58.481 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2023-12-16T21:02:58.481 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2023-12-16T21:02:58.481 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T21:02:58.482 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2023-12-16T21:02:58.482 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2023-12-16T21:02:58.482 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:02:58.482 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T21:02:58.482 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T21:02:58.482 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T21:02:58.483 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2023-12-16T21:02:58.483 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:02:58.483 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T21:02:58.484 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T21:02:58.484 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-391.el9.x86_64 2023-12-16T21:02:58.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T21:02:59.076 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2023-12-16T21:03:00.650 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18218: grub2-probe 2023-12-16T21:03:00.650 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18218: grub2-probe 2023-12-16T21:03:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18218: grub2-probe 2023-12-16T21:03:00.676 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18218: grub2-probe 2023-12-16T21:03:00.861 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18266: grub2-probe 2023-12-16T21:03:00.861 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18266: grub2-probe 2023-12-16T21:03:00.899 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18266: grub2-probe 2023-12-16T21:03:00.899 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18266: grub2-probe 2023-12-16T21:03:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18314: grub2-probe 2023-12-16T21:03:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18314: grub2-probe 2023-12-16T21:03:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18314: grub2-probe 2023-12-16T21:03:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18314: grub2-probe 2023-12-16T21:03:01.237 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18362: grub2-probe 2023-12-16T21:03:01.237 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18362: grub2-probe 2023-12-16T21:03:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18362: grub2-probe 2023-12-16T21:03:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18362: grub2-probe 2023-12-16T21:03:01.410 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18410: grub2-probe 2023-12-16T21:03:01.410 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18410: grub2-probe 2023-12-16T21:03:01.446 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[37950]) leaked on vgs invocation. Parent PID 18410: grub2-probe 2023-12-16T21:03:01.446 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[43869]) leaked on vgs invocation. Parent PID 18410: grub2-probe 2023-12-16T21:03:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T21:03:01.576 INFO:teuthology.orchestra.run.smithi046.stderr:done 2023-12-16T21:03:01.578 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-12-16T21:03:01.593 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.fsA2qwAV5a 2023-12-16T21:03:01.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fsA2qwAV5a 2023-12-16T21:03:01.672 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.fsA2qwAV5a 2023-12-16T21:03:01.818 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.fsA2qwAV5a is 6KB 2023-12-16T21:03:01.866 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.fsA2qwAV5a 2023-12-16T21:03:01.880 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T21:03:01.949 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T21:03:01.950 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T21:03:01.950 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T21:03:01.950 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T21:03:01.951 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T21:03:02.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2023-12-16T21:03:13.857 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-16T21:03:13.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-12-16T21:03:13.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:03:32.121 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-16T21:03:32.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-12-16T21:03:32.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:03:48.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-12-16T21:03:57.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-12-16T21:03:57.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:04:06.677 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-12-16T21:04:15.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-12-16T21:04:15.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:04:15.889 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2023-12-16T21:04:27.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-12-16T21:04:27.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:04:28.313 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-12-16T21:04:28.792 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-12-16T21:04:28.792 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-391.el9.x86_64"... 2023-12-16T21:04:28.792 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-12-16T21:04:28.806 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-391.el9.x86_64 2023-12-16T21:04:28.806 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T21:04:28.806 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T21:04:28.806 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T21:04:29.807 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-16T21:04:29.808 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T21:04:29.885 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2023-12-16T21:04:29.930 DEBUG:teuthology.parallel:result is None 2023-12-16T21:04:34.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2023-12-16T21:04:46.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-12-16T21:04:46.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:04:46.463 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-12-16T21:04:47.038 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-12-16T21:04:47.038 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-16T21:04:47.038 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2023-12-16T21:04:47.053 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-391.el9.x86_64 2023-12-16T21:04:47.053 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T21:04:47.053 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T21:04:47.053 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T21:04:48.055 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-16T21:04:48.055 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T21:04:48.129 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2023-12-16T21:04:48.176 DEBUG:teuthology.parallel:result is None 2023-12-16T21:04:48.177 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-16T21:04:48.184 INFO:teuthology.task.internal:Creating test directory... 2023-12-16T21:04:48.185 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T21:04:48.187 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T21:04:48.205 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-16T21:04:48.226 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-16T21:04:48.259 INFO:teuthology.task.internal:Creating archive directory... 2023-12-16T21:04:48.260 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T21:04:48.262 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T21:04:48.384 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-16T21:04:48.391 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-16T21:04:48.392 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T21:04:48.394 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T21:04:48.431 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:04:48.435 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:04:48.445 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:04:48.452 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:04:48.453 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-16T21:04:48.461 INFO:teuthology.task.internal:Configuring sudo... 2023-12-16T21:04:48.461 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T21:04:48.463 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T21:04:48.519 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-16T21:04:48.528 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-16T21:04:48.529 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:04:48.535 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:04:48.576 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:04:48.635 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:04:48.690 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:04:48.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:04:48.826 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:04:48.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T21:04:48.910 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:04:48.957 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:04:49.018 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:04:49.062 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:04:49.107 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:04:49.107 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T21:04:49.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2023-12-16T21:04:49.179 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2023-12-16T21:04:49.232 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:04:49.269 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:04:49.504 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-16T21:04:49.513 INFO:teuthology.task.internal:Starting timer... 2023-12-16T21:04:49.513 INFO:teuthology.run_tasks:Running task pcp... 2023-12-16T21:04:49.553 INFO:teuthology.run_tasks:Running task selinux... 2023-12-16T21:04:49.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2023-12-16T21:04:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2023-12-16T21:04:49.648 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2023-12-16T21:04:49.689 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2023-12-16T21:04:49.691 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-16T21:04:49.691 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2023-12-16T21:04:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2023-12-16T21:04:49.716 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2023-12-16T21:04:49.760 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2023-12-16T21:04:49.761 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2023-12-16T21:04:49.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T21:04:49.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:04:49.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T21:04:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:04:49.828 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-16T21:04:49.828 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2023-12-16T21:04:49.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2023-12-16T21:04:49.910 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-16T21:04:49.921 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-16T21:04:49.921 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-16T21:04:49.938 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-16T21:04:49.939 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-16T21:10:46.487 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2023-12-16T21:10:46.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-12-16T21:10:46.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:10:46.588 DEBUG:teuthology.orchestra.run.smithi046:> true 2023-12-16T21:10:46.656 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2023-12-16T21:10:46.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-12-16T21:10:46.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:10:46.741 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-12-16T21:10:46.808 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-12-16T21:10:46.809 INFO:teuthology.run_tasks:Running task clock... 2023-12-16T21:10:46.820 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-16T21:10:46.820 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T21:10:46.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:10:46.823 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T21:10:46.823 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:10:46.855 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T21:10:46.875 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T21:10:46.878 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T21:10:46.895 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T21:10:46.911 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2023-12-16T21:10:46.924 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2023-12-16T21:10:46.928 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2023-12-16T21:10:46.937 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2023-12-16T21:10:46.947 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T21:10:46.954 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T21:10:46.967 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T21:10:46.970 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T21:10:47.013 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:10:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:10:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:10:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-12-16T21:10:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.017 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.018 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:10:47.018 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2023-12-16T21:10:47.019 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.019 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.019 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.019 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:10:47.019 INFO:teuthology.run_tasks:Running task install... 2023-12-16T21:10:47.027 DEBUG:teuthology.task.install:project ceph 2023-12-16T21:10:47.028 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}} 2023-12-16T21:10:47.028 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-16T21:10:47.028 INFO:teuthology.task.install:Using flavor: default 2023-12-16T21:10:47.035 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-16T21:10:47.035 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2023-12-16T21:10:47.036 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-16T21:10:47.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:10:47.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-16T21:10:47.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:10:47.187 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/ 2023-12-16T21:10:47.187 INFO:teuthology.task.install.rpm:Package version is 19.0.0-202.gc7586e95 2023-12-16T21:10:47.191 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/ 2023-12-16T21:10:47.191 INFO:teuthology.task.install.rpm:Package version is 19.0.0-202.gc7586e95 2023-12-16T21:10:47.296 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T21:10:47.296 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:10:47.296 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T21:10:47.301 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-yuri8-testing-2023-12-15-0911/c7586e95519b7be76f3b6e3a4b333888baf46bc9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T21:10:47.301 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:10:47.302 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T21:10:47.324 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-12-16T21:10:47.325 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7586e95519b7be76f3b6e3a4b333888baf46bc9/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T21:10:47.334 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-12-16T21:10:47.334 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7586e95519b7be76f3b6e3a4b333888baf46bc9/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T21:10:47.392 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T21:10:47.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T21:10:47.471 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T21:10:47.496 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T21:10:47.526 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2023-12-16T21:10:47.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2023-12-16T21:10:47.538 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2023-12-16T21:10:47.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2023-12-16T21:10:47.777 INFO:teuthology.orchestra.run.smithi046.stdout:73 files removed 2023-12-16T21:10:47.788 INFO:teuthology.orchestra.run.smithi121.stdout:73 files removed 2023-12-16T21:10:47.805 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-12-16T21:10:47.817 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-12-16T21:10:48.527 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 206 kB/s | 78 kB 00:00 2023-12-16T21:10:48.545 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 194 kB/s | 78 kB 00:00 2023-12-16T21:10:48.951 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-16T21:10:48.975 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-16T21:10:49.317 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2023-12-16T21:10:49.360 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2023-12-16T21:10:49.961 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-12-16T21:10:49.996 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-12-16T21:10:51.303 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-12-16T21:10:51.404 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-12-16T21:10:52.052 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 7.9 MB 00:00 2023-12-16T21:10:53.086 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 5.9 MB/s | 7.9 MB 00:01 2023-12-16T21:10:54.069 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 19 MB/s | 18 MB 00:00 2023-12-16T21:10:55.196 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 18 MB/s | 18 MB 00:01 2023-12-16T21:10:59.087 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.8 MB 00:01 2023-12-16T21:11:00.373 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.8 MB 00:01 2023-12-16T21:11:00.789 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2023-12-16T21:11:02.249 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2023-12-16T21:11:03.149 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2023-12-16T21:11:08.049 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2023-12-16T21:11:10.554 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-16T21:11:10.555 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-16T21:11:10.570 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-12-16T21:11:10.571 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-12-16T21:11:10.572 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-12-16T21:11:10.573 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-12-16T21:11:10.575 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-12-16T21:11:10.578 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-12-16T21:11:10.579 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-12-16T21:11:10.659 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2023-12-16T21:11:10.669 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================== 2023-12-16T21:11:10.669 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2023-12-16T21:11:10.669 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================== 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-202.gc7586e95.el9 ceph 6.5 k 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-202.gc7586e95.el9 ceph 5.2 M 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-202.gc7586e95.el9 ceph 861 k 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-202.gc7586e95.el9 ceph 141 k 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-202.gc7586e95.el9 ceph 1.5 M 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 133 k 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 1.8 M 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 7.4 M 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 50 k 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-202.gc7586e95.el9 ceph 8.6 M 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-202.gc7586e95.el9 ceph 41 M 2023-12-16T21:11:10.670 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 262 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 805 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-202.gc7586e95.el9 ceph 217 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-202.gc7586e95.el9 ceph 27 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 18 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-202.gc7586e95.el9 ceph 31 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 719 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-202.gc7586e95.el9 ceph 126 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-12-16T21:11:10.671 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-202.gc7586e95.el9 ceph 162 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-202.gc7586e95.el9 ceph 320 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-202.gc7586e95.el9 ceph 298 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-202.gc7586e95.el9 ceph 99 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-202.gc7586e95.el9 ceph 85 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-202.gc7586e95.el9 ceph 3.0 M 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-202.gc7586e95.el9 ceph 170 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager x86_64 1:1.45.7-1.el9 baseos 2.3 M 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm x86_64 1:1.45.7-1.el9 baseos 1.8 M 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team x86_64 1:1.45.7-1.el9 baseos 43 k 2023-12-16T21:11:10.672 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui x86_64 1:1.45.7-1.el9 baseos 246 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 3.3 M 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 3.0 M 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-12-16T21:11:10.673 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-202.gc7586e95.el9 ceph 19 M 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 28 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-202.gc7586e95.el9 ceph 2.1 M 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 243 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-202.gc7586e95.el9 ceph 4.4 M 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-202.gc7586e95.el9 ceph 18 M 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 15 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-202.gc7586e95.el9 ceph 24 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T21:11:10.674 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-202.gc7586e95.el9 ceph 164 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 476 k 2023-12-16T21:11:10.675 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 4.8 M 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T21:11:10.676 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-202.gc7586e95.el9 ceph 45 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-202.gc7586e95.el9 ceph 130 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2023-12-16T21:11:10.677 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T21:11:10.678 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T21:11:10.679 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T21:11:10.680 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================== 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout:Install 140 Packages 2023-12-16T21:11:10.681 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 15 Packages 2023-12-16T21:11:10.682 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:10.682 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 205 M 2023-12-16T21:11:10.682 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2023-12-16T21:11:11.212 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 1.9 MB/s | 20 MB 00:10 2023-12-16T21:11:11.358 INFO:teuthology.orchestra.run.smithi046.stdout:(1/155): ceph-19.0.0-202.gc7586e95.el9.x86_64.r 41 kB/s | 6.5 kB 00:00 2023-12-16T21:11:11.559 INFO:teuthology.orchestra.run.smithi046.stdout:(2/155): ceph-fuse-19.0.0-202.gc7586e95.el9.x86 4.2 MB/s | 861 kB 00:00 2023-12-16T21:11:11.609 INFO:teuthology.orchestra.run.smithi046.stdout:(3/155): ceph-immutable-object-cache-19.0.0-202 2.8 MB/s | 141 kB 00:00 2023-12-16T21:11:11.859 INFO:teuthology.orchestra.run.smithi046.stdout:(4/155): ceph-mds-19.0.0-202.gc7586e95.el9.x86_ 8.4 MB/s | 2.1 MB 00:00 2023-12-16T21:11:12.068 INFO:teuthology.orchestra.run.smithi046.stdout:(5/155): ceph-mgr-19.0.0-202.gc7586e95.el9.x86_ 7.0 MB/s | 1.5 MB 00:00 2023-12-16T21:11:12.151 INFO:teuthology.orchestra.run.smithi046.stdout:(6/155): ceph-base-19.0.0-202.gc7586e95.el9.x86 5.5 MB/s | 5.2 MB 00:00 2023-12-16T21:11:12.593 INFO:teuthology.orchestra.run.smithi046.stdout:(7/155): ceph-mon-19.0.0-202.gc7586e95.el9.x86_ 8.4 MB/s | 4.4 MB 00:00 2023-12-16T21:11:13.735 INFO:teuthology.orchestra.run.smithi046.stdout:(8/155): ceph-osd-19.0.0-202.gc7586e95.el9.x86_ 11 MB/s | 18 MB 00:01 2023-12-16T21:11:13.827 INFO:teuthology.orchestra.run.smithi046.stdout:(9/155): ceph-radosgw-19.0.0-202.gc7586e95.el9. 7.0 MB/s | 8.6 MB 00:01 2023-12-16T21:11:13.852 INFO:teuthology.orchestra.run.smithi046.stdout:(10/155): ceph-selinux-19.0.0-202.gc7586e95.el9 206 kB/s | 24 kB 00:00 2023-12-16T21:11:14.052 INFO:teuthology.orchestra.run.smithi046.stdout:(11/155): ceph-common-19.0.0-202.gc7586e95.el9. 6.7 MB/s | 19 MB 00:02 2023-12-16T21:11:14.086 INFO:teuthology.orchestra.run.smithi046.stdout:(12/155): cephfs-mirror-19.0.0-202.gc7586e95.el 930 kB/s | 217 kB 00:00 2023-12-16T21:11:14.111 INFO:teuthology.orchestra.run.smithi046.stdout:(13/155): cephfs-shell-19.0.0-202.gc7586e95.el9 461 kB/s | 27 kB 00:00 2023-12-16T21:11:14.153 INFO:teuthology.orchestra.run.smithi046.stdout:(14/155): libcephfs-devel-19.0.0-202.gc7586e95. 465 kB/s | 31 kB 00:00 2023-12-16T21:11:14.228 INFO:teuthology.orchestra.run.smithi046.stdout:(15/155): libcephsqlite-19.0.0-202.gc7586e95.el 2.1 MB/s | 164 kB 00:00 2023-12-16T21:11:14.262 INFO:teuthology.orchestra.run.smithi046.stdout:(16/155): libcephfs2-19.0.0-202.gc7586e95.el9.x 4.7 MB/s | 719 kB 00:00 2023-12-16T21:11:14.296 INFO:teuthology.orchestra.run.smithi046.stdout:(17/155): librados-devel-19.0.0-202.gc7586e95.e 1.8 MB/s | 126 kB 00:00 2023-12-16T21:11:14.346 INFO:teuthology.orchestra.run.smithi046.stdout:(18/155): libradosstriper1-19.0.0-202.gc7586e95 5.6 MB/s | 476 kB 00:00 2023-12-16T21:11:14.421 INFO:teuthology.orchestra.run.smithi046.stdout:(19/155): python3-ceph-argparse-19.0.0-202.gc75 601 kB/s | 45 kB 00:00 2023-12-16T21:11:14.522 INFO:teuthology.orchestra.run.smithi046.stdout:(20/155): python3-ceph-common-19.0.0-202.gc7586 1.3 MB/s | 130 kB 00:00 2023-12-16T21:11:14.615 INFO:teuthology.orchestra.run.smithi046.stdout:(21/155): python3-cephfs-19.0.0-202.gc7586e95.e 1.7 MB/s | 162 kB 00:00 2023-12-16T21:11:14.690 INFO:teuthology.orchestra.run.smithi046.stdout:(22/155): librgw2-19.0.0-202.gc7586e95.el9.x86_ 12 MB/s | 4.8 MB 00:00 2023-12-16T21:11:14.716 INFO:teuthology.orchestra.run.smithi046.stdout:(23/155): python3-rados-19.0.0-202.gc7586e95.el 3.1 MB/s | 320 kB 00:00 2023-12-16T21:11:14.757 INFO:teuthology.orchestra.run.smithi046.stdout:(24/155): python3-rbd-19.0.0-202.gc7586e95.el9. 4.3 MB/s | 298 kB 00:00 2023-12-16T21:11:14.783 INFO:teuthology.orchestra.run.smithi046.stdout:(25/155): python3-rgw-19.0.0-202.gc7586e95.el9. 1.4 MB/s | 99 kB 00:00 2023-12-16T21:11:14.825 INFO:teuthology.orchestra.run.smithi046.stdout:(26/155): rbd-fuse-19.0.0-202.gc7586e95.el9.x86 1.2 MB/s | 85 kB 00:00 2023-12-16T21:11:14.900 INFO:teuthology.orchestra.run.smithi046.stdout:(27/155): rbd-nbd-19.0.0-202.gc7586e95.el9.x86_ 2.2 MB/s | 170 kB 00:00 2023-12-16T21:11:14.983 INFO:teuthology.orchestra.run.smithi046.stdout:(28/155): ceph-grafana-dashboards-19.0.0-202.gc 338 kB/s | 28 kB 00:00 2023-12-16T21:11:15.059 INFO:teuthology.orchestra.run.smithi046.stdout:(29/155): ceph-mgr-cephadm-19.0.0-202.gc7586e95 1.7 MB/s | 133 kB 00:00 2023-12-16T21:11:15.159 INFO:teuthology.orchestra.run.smithi046.stdout:(30/155): rbd-mirror-19.0.0-202.gc7586e95.el9.x 8.0 MB/s | 3.0 MB 00:00 2023-12-16T21:11:15.234 INFO:teuthology.orchestra.run.smithi046.stdout:(31/155): ceph-mgr-dashboard-19.0.0-202.gc7586e 10 MB/s | 1.8 MB 00:00 2023-12-16T21:11:15.318 INFO:teuthology.orchestra.run.smithi046.stdout:(32/155): ceph-mgr-modules-core-19.0.0-202.gc75 2.9 MB/s | 243 kB 00:00 2023-12-16T21:11:15.385 INFO:teuthology.orchestra.run.smithi046.stdout:(33/155): ceph-mgr-rook-19.0.0-202.gc7586e95.el 754 kB/s | 50 kB 00:00 2023-12-16T21:11:15.460 INFO:teuthology.orchestra.run.smithi046.stdout:(34/155): ceph-prometheus-alerts-19.0.0-202.gc7 202 kB/s | 15 kB 00:00 2023-12-16T21:11:15.544 INFO:teuthology.orchestra.run.smithi046.stdout:(35/155): ceph-volume-19.0.0-202.gc7586e95.el9. 3.1 MB/s | 262 kB 00:00 2023-12-16T21:11:15.644 INFO:teuthology.orchestra.run.smithi046.stdout:(36/155): cephadm-19.0.0-202.gc7586e95.el9.noar 7.9 MB/s | 805 kB 00:00 2023-12-16T21:11:15.727 INFO:teuthology.orchestra.run.smithi046.stdout:(37/155): cephfs-top-19.0.0-202.gc7586e95.el9.n 220 kB/s | 18 kB 00:00 2023-12-16T21:11:15.853 INFO:teuthology.orchestra.run.smithi046.stdout:(38/155): automake-1.16.2-8.el9.noarch.rpm 5.2 MB/s | 662 kB 00:00 2023-12-16T21:11:15.911 INFO:teuthology.orchestra.run.smithi046.stdout:(39/155): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2023-12-16T21:11:15.945 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2023-12-16T21:11:16.003 INFO:teuthology.orchestra.run.smithi046.stdout:(40/155): ceph-mgr-diskprediction-local-19.0.0- 8.8 MB/s | 7.4 MB 00:00 2023-12-16T21:11:16.029 INFO:teuthology.orchestra.run.smithi046.stdout:(41/155): boost-program-options-1.75.0-8.el9.x8 904 kB/s | 106 kB 00:00 2023-12-16T21:11:16.054 INFO:teuthology.orchestra.run.smithi046.stdout:(42/155): elfutils-libelf-devel-0.190-1.el9.x86 432 kB/s | 21 kB 00:00 2023-12-16T21:11:16.079 INFO:teuthology.orchestra.run.smithi046.stdout:(43/155): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2023-12-16T21:11:16.104 INFO:teuthology.orchestra.run.smithi046.stdout:(44/155): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2023-12-16T21:11:16.172 INFO:teuthology.orchestra.run.smithi046.stdout:(45/155): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2023-12-16T21:11:16.197 INFO:teuthology.orchestra.run.smithi046.stdout:(46/155): flexiblas-openblas-openmp-3.0.4-8.el9 192 kB/s | 17 kB 00:00 2023-12-16T21:11:16.222 INFO:teuthology.orchestra.run.smithi046.stdout:(47/155): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2023-12-16T21:11:16.247 INFO:teuthology.orchestra.run.smithi046.stdout:(48/155): libaio-devel-0.3.111-13.el9.x86_64.rp 203 kB/s | 10 kB 00:00 2023-12-16T21:11:16.273 INFO:teuthology.orchestra.run.smithi046.stdout:(49/155): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-16T21:11:16.298 INFO:teuthology.orchestra.run.smithi046.stdout:(50/155): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2023-12-16T21:11:16.323 INFO:teuthology.orchestra.run.smithi046.stdout:(51/155): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-16T21:11:16.349 INFO:teuthology.orchestra.run.smithi046.stdout:(52/155): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2023-12-16T21:11:16.374 INFO:teuthology.orchestra.run.smithi046.stdout:(53/155): libuuid-devel-2.37.4-15.el9.x86_64.rp 498 kB/s | 24 kB 00:00 2023-12-16T21:11:16.400 INFO:teuthology.orchestra.run.smithi046.stdout:(54/155): libzstd-devel-1.5.1-2.el9.x86_64.rpm 927 kB/s | 46 kB 00:00 2023-12-16T21:11:16.425 INFO:teuthology.orchestra.run.smithi046.stdout:(55/155): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-12-16T21:11:16.450 INFO:teuthology.orchestra.run.smithi046.stdout:(56/155): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-12-16T21:11:16.475 INFO:teuthology.orchestra.run.smithi046.stdout:(57/155): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-16T21:11:16.492 INFO:teuthology.orchestra.run.smithi046.stdout:(58/155): openblas-0.3.21-2.el9.x86_64.rpm 830 kB/s | 34 kB 00:00 2023-12-16T21:11:16.559 INFO:teuthology.orchestra.run.smithi046.stdout:(59/155): openblas-openmp-0.3.21-2.el9.x86_64.r 57 MB/s | 4.7 MB 00:00 2023-12-16T21:11:16.601 INFO:teuthology.orchestra.run.smithi046.stdout:(60/155): postgresql-13.11-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2023-12-16T21:11:16.635 INFO:teuthology.orchestra.run.smithi046.stdout:(61/155): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2023-12-16T21:11:16.660 INFO:teuthology.orchestra.run.smithi046.stdout:(62/155): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2023-12-16T21:11:16.743 INFO:teuthology.orchestra.run.smithi046.stdout:(63/155): postgresql-server-13.11-1.el9.x86_64. 53 MB/s | 5.7 MB 00:00 2023-12-16T21:11:16.760 INFO:teuthology.orchestra.run.smithi046.stdout:(64/155): python3-attrs-20.3.0-7.el9.noarch.rpm 800 kB/s | 78 kB 00:00 2023-12-16T21:11:16.794 INFO:teuthology.orchestra.run.smithi046.stdout:(65/155): python3-jinja2-2.11.3-4.el9.noarch.rp 6.8 MB/s | 228 kB 00:00 2023-12-16T21:11:16.877 INFO:teuthology.orchestra.run.smithi046.stdout:(66/155): python3-babel-2.9.1-2.el9.noarch.rpm 50 MB/s | 5.8 MB 00:00 2023-12-16T21:11:16.902 INFO:teuthology.orchestra.run.smithi046.stdout:(67/155): python3-jsonpatch-1.21-16.el9.noarch. 232 kB/s | 24 kB 00:00 2023-12-16T21:11:16.936 INFO:teuthology.orchestra.run.smithi046.stdout:(68/155): python3-jsonpointer-2.0-4.el9.noarch. 545 kB/s | 18 kB 00:00 2023-12-16T21:11:16.962 INFO:teuthology.orchestra.run.smithi046.stdout:(69/155): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2023-12-16T21:11:16.987 INFO:teuthology.orchestra.run.smithi046.stdout:(70/155): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2023-12-16T21:11:17.054 INFO:teuthology.orchestra.run.smithi046.stdout:(71/155): python3-numpy-1.20.1-5.el9.x86_64.rpm 52 MB/s | 4.8 MB 00:00 2023-12-16T21:11:17.087 INFO:teuthology.orchestra.run.smithi046.stdout:(72/155): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.2 MB/s | 217 kB 00:00 2023-12-16T21:11:17.112 INFO:teuthology.orchestra.run.smithi046.stdout:(73/155): python3-oauthlib-3.1.1-5.el9.noarch.r 3.0 MB/s | 174 kB 00:00 2023-12-16T21:11:17.138 INFO:teuthology.orchestra.run.smithi046.stdout:(74/155): python3-prettytable-0.7.2-27.el9.noar 827 kB/s | 41 kB 00:00 2023-12-16T21:11:17.163 INFO:teuthology.orchestra.run.smithi046.stdout:(75/155): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-12-16T21:11:17.188 INFO:teuthology.orchestra.run.smithi046.stdout:(76/155): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2023-12-16T21:11:17.213 INFO:teuthology.orchestra.run.smithi046.stdout:(77/155): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2023-12-16T21:11:17.239 INFO:teuthology.orchestra.run.smithi046.stdout:(78/155): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2023-12-16T21:11:17.331 INFO:teuthology.orchestra.run.smithi046.stdout:(79/155): python3-toml-0.10.2-6.el9.noarch.rpm 437 kB/s | 39 kB 00:00 2023-12-16T21:11:17.481 INFO:teuthology.orchestra.run.smithi046.stdout:(80/155): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-12-16T21:11:17.506 INFO:teuthology.orchestra.run.smithi046.stdout:(81/155): python3-wcwidth-0.2.5-8.el9.noarch.rp 237 kB/s | 41 kB 00:00 2023-12-16T21:11:17.531 INFO:teuthology.orchestra.run.smithi046.stdout:(82/155): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-12-16T21:11:17.557 INFO:teuthology.orchestra.run.smithi046.stdout:(83/155): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2023-12-16T21:11:17.585 INFO:teuthology.orchestra.run.smithi046.stdout:(84/155): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.7 MB/s | 85 kB 00:00 2023-12-16T21:11:17.607 INFO:teuthology.orchestra.run.smithi046.stdout:(85/155): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-16T21:11:17.633 INFO:teuthology.orchestra.run.smithi046.stdout:(86/155): zlib-devel-1.2.11-40.el9.x86_64.rpm 876 kB/s | 44 kB 00:00 2023-12-16T21:11:17.658 INFO:teuthology.orchestra.run.smithi046.stdout:(87/155): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2023-12-16T21:11:17.700 INFO:teuthology.orchestra.run.smithi046.stdout:(88/155): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-12-16T21:11:17.725 INFO:teuthology.orchestra.run.smithi046.stdout:(89/155): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2023-12-16T21:11:17.751 INFO:teuthology.orchestra.run.smithi046.stdout:(90/155): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-16T21:11:17.784 INFO:teuthology.orchestra.run.smithi046.stdout:(91/155): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2023-12-16T21:11:17.818 INFO:teuthology.orchestra.run.smithi046.stdout:(92/155): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2023-12-16T21:11:17.843 INFO:teuthology.orchestra.run.smithi046.stdout:(93/155): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2023-12-16T21:11:17.868 INFO:teuthology.orchestra.run.smithi046.stdout:(94/155): python3-requests-2.25.1-7.el9.noarch. 2.2 MB/s | 113 kB 00:00 2023-12-16T21:11:17.893 INFO:teuthology.orchestra.run.smithi046.stdout:(95/155): python3-urllib3-1.26.5-4.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2023-12-16T21:11:18.294 INFO:teuthology.orchestra.run.smithi046.stdout:(96/155): ceph-test-19.0.0-202.gc7586e95.el9.x8 9.2 MB/s | 41 MB 00:04 2023-12-16T21:11:18.353 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-16T21:11:18.354 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-16T21:11:18.367 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-12-16T21:11:18.368 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-12-16T21:11:18.369 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-12-16T21:11:18.370 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-12-16T21:11:18.372 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-12-16T21:11:18.375 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-12-16T21:11:18.376 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-12-16T21:11:18.419 INFO:teuthology.orchestra.run.smithi046.stdout:(97/155): gperftools-libs-2.9.1-2.el9.x86_64.rp 2.4 MB/s | 309 kB 00:00 2023-12-16T21:11:18.450 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-12-16T21:11:18.471 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================== 2023-12-16T21:11:18.471 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-12-16T21:11:18.471 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================== 2023-12-16T21:11:18.471 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-202.gc7586e95.el9 ceph 6.5 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-202.gc7586e95.el9 ceph 5.2 M 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-202.gc7586e95.el9 ceph 861 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-202.gc7586e95.el9 ceph 141 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-202.gc7586e95.el9 ceph 1.5 M 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 133 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 1.8 M 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 7.4 M 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 50 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-202.gc7586e95.el9 ceph 8.6 M 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-202.gc7586e95.el9 ceph 41 M 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 262 k 2023-12-16T21:11:18.472 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 805 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:19.0.0-202.gc7586e95.el9 ceph 217 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell x86_64 2:19.0.0-202.gc7586e95.el9 ceph 27 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 18 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-202.gc7586e95.el9 ceph 31 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 719 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-202.gc7586e95.el9 ceph 126 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-12-16T21:11:18.473 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-202.gc7586e95.el9 ceph 162 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-202.gc7586e95.el9 ceph 320 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-202.gc7586e95.el9 ceph 298 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-202.gc7586e95.el9 ceph 99 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-202.gc7586e95.el9 ceph 85 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-202.gc7586e95.el9 ceph 3.0 M 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-202.gc7586e95.el9 ceph 170 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager x86_64 1:1.45.7-1.el9 baseos 2.3 M 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-libnm x86_64 1:1.45.7-1.el9 baseos 1.8 M 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-team x86_64 1:1.45.7-1.el9 baseos 43 k 2023-12-16T21:11:18.474 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-tui x86_64 1:1.45.7-1.el9 baseos 246 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 3.3 M 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 3.0 M 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-12-16T21:11:18.475 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-202.gc7586e95.el9 ceph 19 M 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 28 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-202.gc7586e95.el9 ceph 2.1 M 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 243 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-202.gc7586e95.el9 ceph 4.4 M 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-202.gc7586e95.el9 ceph 18 M 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-202.gc7586e95.el9 ceph-noarch 15 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-202.gc7586e95.el9 ceph 24 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T21:11:18.476 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-202.gc7586e95.el9 ceph 164 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 476 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-202.gc7586e95.el9 ceph 4.8 M 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-16T21:11:18.477 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.0.0-202.gc7586e95.el9 ceph 45 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-202.gc7586e95.el9 ceph 130 k 2023-12-16T21:11:18.478 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T21:11:18.479 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T21:11:18.480 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T21:11:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T21:11:18.482 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================== 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:Install 140 Packages 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 15 Packages 2023-12-16T21:11:18.483 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:11:18.484 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 205 M 2023-12-16T21:11:18.484 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-12-16T21:11:18.619 INFO:teuthology.orchestra.run.smithi046.stdout:(98/155): libarrow-9.0.0-10.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2023-12-16T21:11:18.653 INFO:teuthology.orchestra.run.smithi046.stdout:(99/155): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2023-12-16T21:11:18.686 INFO:teuthology.orchestra.run.smithi046.stdout:(100/155): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2023-12-16T21:11:18.729 INFO:teuthology.orchestra.run.smithi046.stdout:(101/155): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2023-12-16T21:11:18.770 INFO:teuthology.orchestra.run.smithi046.stdout:(102/155): python3-devel-3.9.17-2.el9.x86_64.rp 280 kB/s | 245 kB 00:00 2023-12-16T21:11:18.796 INFO:teuthology.orchestra.run.smithi046.stdout:(103/155): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2023-12-16T21:11:18.854 INFO:teuthology.orchestra.run.smithi046.stdout:(104/155): python3-asyncssh-2.13.2-3.el9.noarch 9.2 MB/s | 548 kB 00:00 2023-12-16T21:11:18.896 INFO:teuthology.orchestra.run.smithi046.stdout:(105/155): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2023-12-16T21:11:18.955 INFO:teuthology.orchestra.run.smithi046.stdout:(106/155): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2023-12-16T21:11:19.005 INFO:teuthology.orchestra.run.smithi046.stdout:(107/155): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2023-12-16T21:11:19.026 INFO:teuthology.orchestra.run.smithi046.stdout:(108/155): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2023-12-16T21:11:19.047 INFO:teuthology.orchestra.run.smithi046.stdout:(109/155): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2023-12-16T21:11:19.073 INFO:teuthology.orchestra.run.smithi046.stdout:(110/155): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2023-12-16T21:11:19.098 INFO:teuthology.orchestra.run.smithi046.stdout:(111/155): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2023-12-16T21:11:19.115 INFO:teuthology.orchestra.run.smithi046.stdout:(112/155): python3-colorama-0.4.4-12.el9.noarch 842 kB/s | 35 kB 00:00 2023-12-16T21:11:19.140 INFO:teuthology.orchestra.run.smithi046.stdout:(113/155): python3-google-auth-2.23.4-1.el9.noa 4.6 MB/s | 197 kB 00:00 2023-12-16T21:11:19.166 INFO:teuthology.orchestra.run.smithi046.stdout:(114/155): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2023-12-16T21:11:19.191 INFO:teuthology.orchestra.run.smithi046.stdout:(115/155): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2023-12-16T21:11:19.195 INFO:teuthology.orchestra.run.smithi121.stdout:(1/155): ceph-19.0.0-202.gc7586e95.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2023-12-16T21:11:19.216 INFO:teuthology.orchestra.run.smithi046.stdout:(116/155): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2023-12-16T21:11:19.241 INFO:teuthology.orchestra.run.smithi046.stdout:(117/155): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2023-12-16T21:11:19.267 INFO:teuthology.orchestra.run.smithi046.stdout:(118/155): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2023-12-16T21:11:19.292 INFO:teuthology.orchestra.run.smithi046.stdout:(119/155): python3-jwt+crypto-2.4.0-1.el9.noarc 180 kB/s | 9.0 kB 00:00 2023-12-16T21:11:19.317 INFO:teuthology.orchestra.run.smithi046.stdout:(120/155): python3-jwt-2.4.0-1.el9.noarch.rpm 814 kB/s | 41 kB 00:00 2023-12-16T21:11:19.376 INFO:teuthology.orchestra.run.smithi046.stdout:(121/155): openssl-devel-3.0.7-25.el9.x86_64.rp 2.7 MB/s | 4.1 MB 00:01 2023-12-16T21:11:19.401 INFO:teuthology.orchestra.run.smithi046.stdout:(122/155): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2023-12-16T21:11:19.444 INFO:teuthology.orchestra.run.smithi046.stdout:(123/155): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2023-12-16T21:11:19.468 INFO:teuthology.orchestra.run.smithi046.stdout:(124/155): python3-natsort-7.1.1-5.el9.noarch.r 866 kB/s | 58 kB 00:00 2023-12-16T21:11:19.494 INFO:teuthology.orchestra.run.smithi046.stdout:(125/155): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2023-12-16T21:11:19.519 INFO:teuthology.orchestra.run.smithi046.stdout:(126/155): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2023-12-16T21:11:19.544 INFO:teuthology.orchestra.run.smithi046.stdout:(127/155): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2023-12-16T21:11:19.569 INFO:teuthology.orchestra.run.smithi046.stdout:(128/155): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2023-12-16T21:11:19.595 INFO:teuthology.orchestra.run.smithi046.stdout:(129/155): python3-pyOpenSSL-21.0.0-1.el9.noarc 894 kB/s | 90 kB 00:00 2023-12-16T21:11:19.753 INFO:teuthology.orchestra.run.smithi121.stdout:(2/155): ceph-fuse-19.0.0-202.gc7586e95.el9.x86 1.6 MB/s | 861 kB 00:00 2023-12-16T21:11:19.754 INFO:teuthology.orchestra.run.smithi046.stdout:(130/155): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2023-12-16T21:11:19.754 INFO:teuthology.orchestra.run.smithi046.stdout:(131/155): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2023-12-16T21:11:19.754 INFO:teuthology.orchestra.run.smithi046.stdout:(132/155): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2023-12-16T21:11:19.754 INFO:teuthology.orchestra.run.smithi046.stdout:(133/155): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2023-12-16T21:11:19.754 INFO:teuthology.orchestra.run.smithi046.stdout:(134/155): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2023-12-16T21:11:19.754 INFO:teuthology.orchestra.run.smithi046.stdout:(135/155): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2023-12-16T21:11:19.772 INFO:teuthology.orchestra.run.smithi046.stdout:(136/155): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2023-12-16T21:11:19.798 INFO:teuthology.orchestra.run.smithi046.stdout:(137/155): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2023-12-16T21:11:19.821 INFO:teuthology.orchestra.run.smithi121.stdout:(3/155): ceph-immutable-object-cache-19.0.0-202 1.5 MB/s | 141 kB 00:00 2023-12-16T21:11:19.832 INFO:teuthology.orchestra.run.smithi046.stdout:(138/155): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2023-12-16T21:11:19.864 INFO:teuthology.orchestra.run.smithi046.stdout:(139/155): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2023-12-16T21:11:19.939 INFO:teuthology.orchestra.run.smithi046.stdout:(140/155): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-12-16T21:11:19.973 INFO:teuthology.orchestra.run.smithi046.stdout:(141/155): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2023-12-16T21:11:20.007 INFO:teuthology.orchestra.run.smithi046.stdout:(142/155): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2023-12-16T21:11:20.038 INFO:teuthology.orchestra.run.smithi121.stdout:(4/155): ceph-base-19.0.0-202.gc7586e95.el9.x86 5.0 MB/s | 5.2 MB 00:01 2023-12-16T21:11:20.040 INFO:teuthology.orchestra.run.smithi046.stdout:(143/155): elfutils-debuginfod-client-0.190-1.e 1.1 MB/s | 36 kB 00:00 2023-12-16T21:11:20.073 INFO:teuthology.orchestra.run.smithi046.stdout:(144/155): elfutils-libelf-0.190-1.el9.x86_64.r 5.7 MB/s | 192 kB 00:00 2023-12-16T21:11:20.107 INFO:teuthology.orchestra.run.smithi046.stdout:(145/155): elfutils-libs-0.190-1.el9.x86_64.rpm 7.5 MB/s | 255 kB 00:00 2023-12-16T21:11:20.141 INFO:teuthology.orchestra.run.smithi046.stdout:(146/155): libacl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 22 kB 00:00 2023-12-16T21:11:20.174 INFO:teuthology.orchestra.run.smithi046.stdout:(147/155): xfsdump-3.1.12-4.el9.x86_64.rpm 9.4 MB/s | 321 kB 00:00 2023-12-16T21:11:20.380 INFO:teuthology.orchestra.run.smithi121.stdout:(5/155): ceph-common-19.0.0-202.gc7586e95.el9.x 14 MB/s | 19 MB 00:01 2023-12-16T21:11:20.416 INFO:teuthology.orchestra.run.smithi046.stdout:(148/155): NetworkManager-1.45.7-1.el9.x86_64.r 9.3 MB/s | 2.3 MB 00:00 2023-12-16T21:11:20.421 INFO:teuthology.orchestra.run.smithi121.stdout:(6/155): ceph-mgr-19.0.0-202.gc7586e95.el9.x86_ 3.8 MB/s | 1.5 MB 00:00 2023-12-16T21:11:20.467 INFO:teuthology.orchestra.run.smithi046.stdout:(149/155): librbd1-19.0.0-202.gc7586e95.el9.x86 5.0 MB/s | 3.0 MB 00:00 2023-12-16T21:11:20.517 INFO:teuthology.orchestra.run.smithi046.stdout:(150/155): librados2-19.0.0-202.gc7586e95.el9.x 4.8 MB/s | 3.3 MB 00:00 2023-12-16T21:11:20.559 INFO:teuthology.orchestra.run.smithi046.stdout:(151/155): NetworkManager-libnm-1.45.7-1.el9.x8 13 MB/s | 1.8 MB 00:00 2023-12-16T21:11:20.592 INFO:teuthology.orchestra.run.smithi046.stdout:(152/155): NetworkManager-team-1.45.7-1.el9.x86 345 kB/s | 43 kB 00:00 2023-12-16T21:11:20.626 INFO:teuthology.orchestra.run.smithi046.stdout:(153/155): openssl-3.0.7-25.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2023-12-16T21:11:20.639 INFO:teuthology.orchestra.run.smithi121.stdout:(7/155): ceph-mds-19.0.0-202.gc7586e95.el9.x86_ 2.6 MB/s | 2.1 MB 00:00 2023-12-16T21:11:20.701 INFO:teuthology.orchestra.run.smithi046.stdout:(154/155): NetworkManager-tui-1.45.7-1.el9.x86_ 1.3 MB/s | 246 kB 00:00 2023-12-16T21:11:20.801 INFO:teuthology.orchestra.run.smithi046.stdout:(155/155): openssl-libs-3.0.7-25.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2023-12-16T21:11:20.806 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:11:20.806 INFO:teuthology.orchestra.run.smithi046.stdout:Total 20 MB/s | 205 MB 00:10 2023-12-16T21:11:20.914 INFO:teuthology.orchestra.run.smithi121.stdout:(8/155): ceph-mon-19.0.0-202.gc7586e95.el9.x86_ 8.2 MB/s | 4.4 MB 00:00 2023-12-16T21:11:20.969 INFO:teuthology.orchestra.run.smithi121.stdout:(9/155): ceph-selinux-19.0.0-202.gc7586e95.el9. 444 kB/s | 24 kB 00:00 2023-12-16T21:11:20.979 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2023-12-16T21:11:21.184 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2023-12-16T21:11:21.184 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2023-12-16T21:11:21.727 INFO:teuthology.orchestra.run.smithi121.stdout:(10/155): ceph-osd-19.0.0-202.gc7586e95.el9.x86 14 MB/s | 18 MB 00:01 2023-12-16T21:11:21.819 INFO:teuthology.orchestra.run.smithi121.stdout:(11/155): cephfs-mirror-19.0.0-202.gc7586e95.el 2.3 MB/s | 217 kB 00:00 2023-12-16T21:11:21.903 INFO:teuthology.orchestra.run.smithi121.stdout:(12/155): cephfs-shell-19.0.0-202.gc7586e95.el9 324 kB/s | 27 kB 00:00 2023-12-16T21:11:21.970 INFO:teuthology.orchestra.run.smithi121.stdout:(13/155): libcephfs-devel-19.0.0-202.gc7586e95. 464 kB/s | 31 kB 00:00 2023-12-16T21:11:22.079 INFO:teuthology.orchestra.run.smithi121.stdout:(14/155): libcephfs2-19.0.0-202.gc7586e95.el9.x 6.5 MB/s | 719 kB 00:00 2023-12-16T21:11:22.171 INFO:teuthology.orchestra.run.smithi121.stdout:(15/155): ceph-radosgw-19.0.0-202.gc7586e95.el9 5.6 MB/s | 8.6 MB 00:01 2023-12-16T21:11:22.196 INFO:teuthology.orchestra.run.smithi121.stdout:(16/155): libcephsqlite-19.0.0-202.gc7586e95.el 1.4 MB/s | 164 kB 00:00 2023-12-16T21:11:22.246 INFO:teuthology.orchestra.run.smithi121.stdout:(17/155): librados-devel-19.0.0-202.gc7586e95.e 1.7 MB/s | 126 kB 00:00 2023-12-16T21:11:22.280 INFO:teuthology.orchestra.run.smithi121.stdout:(18/155): libradosstriper1-19.0.0-202.gc7586e95 5.6 MB/s | 476 kB 00:00 2023-12-16T21:11:22.356 INFO:teuthology.orchestra.run.smithi121.stdout:(19/155): python3-ceph-argparse-19.0.0-202.gc75 603 kB/s | 45 kB 00:00 2023-12-16T21:11:22.439 INFO:teuthology.orchestra.run.smithi121.stdout:(20/155): python3-ceph-common-19.0.0-202.gc7586 1.5 MB/s | 130 kB 00:00 2023-12-16T21:11:22.522 INFO:teuthology.orchestra.run.smithi121.stdout:(21/155): python3-cephfs-19.0.0-202.gc7586e95.e 1.9 MB/s | 162 kB 00:00 2023-12-16T21:11:22.638 INFO:teuthology.orchestra.run.smithi121.stdout:(22/155): python3-rados-19.0.0-202.gc7586e95.el 2.7 MB/s | 320 kB 00:00 2023-12-16T21:11:22.725 INFO:teuthology.orchestra.run.smithi121.stdout:(23/155): librgw2-19.0.0-202.gc7586e95.el9.x86_ 10 MB/s | 4.8 MB 00:00 2023-12-16T21:11:22.748 INFO:teuthology.orchestra.run.smithi121.stdout:(24/155): python3-rbd-19.0.0-202.gc7586e95.el9. 2.7 MB/s | 298 kB 00:00 2023-12-16T21:11:22.798 INFO:teuthology.orchestra.run.smithi121.stdout:(25/155): python3-rgw-19.0.0-202.gc7586e95.el9. 1.3 MB/s | 99 kB 00:00 2023-12-16T21:11:22.823 INFO:teuthology.orchestra.run.smithi121.stdout:(26/155): rbd-fuse-19.0.0-202.gc7586e95.el9.x86 1.1 MB/s | 85 kB 00:00 2023-12-16T21:11:22.933 INFO:teuthology.orchestra.run.smithi121.stdout:(27/155): rbd-nbd-19.0.0-202.gc7586e95.el9.x86_ 1.5 MB/s | 170 kB 00:00 2023-12-16T21:11:23.024 INFO:teuthology.orchestra.run.smithi121.stdout:(28/155): ceph-grafana-dashboards-19.0.0-202.gc 307 kB/s | 28 kB 00:00 2023-12-16T21:11:23.099 INFO:teuthology.orchestra.run.smithi121.stdout:(29/155): rbd-mirror-19.0.0-202.gc7586e95.el9.x 10 MB/s | 3.0 MB 00:00 2023-12-16T21:11:23.125 INFO:teuthology.orchestra.run.smithi121.stdout:(30/155): ceph-mgr-cephadm-19.0.0-202.gc7586e95 1.3 MB/s | 133 kB 00:00 2023-12-16T21:11:23.203 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2023-12-16T21:11:23.203 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2023-12-16T21:11:23.292 INFO:teuthology.orchestra.run.smithi121.stdout:(31/155): ceph-mgr-dashboard-19.0.0-202.gc7586e 9.2 MB/s | 1.8 MB 00:00 2023-12-16T21:11:23.375 INFO:teuthology.orchestra.run.smithi121.stdout:(32/155): ceph-mgr-modules-core-19.0.0-202.gc75 2.9 MB/s | 243 kB 00:00 2023-12-16T21:11:23.534 INFO:teuthology.orchestra.run.smithi121.stdout:(33/155): ceph-mgr-rook-19.0.0-202.gc7586e95.el 317 kB/s | 50 kB 00:00 2023-12-16T21:11:23.659 INFO:teuthology.orchestra.run.smithi121.stdout:(34/155): ceph-prometheus-alerts-19.0.0-202.gc7 121 kB/s | 15 kB 00:00 2023-12-16T21:11:23.751 INFO:teuthology.orchestra.run.smithi121.stdout:(35/155): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2023-12-16T21:11:23.776 INFO:teuthology.orchestra.run.smithi121.stdout:(36/155): ceph-volume-19.0.0-202.gc7586e95.el9. 2.2 MB/s | 262 kB 00:00 2023-12-16T21:11:23.852 INFO:teuthology.orchestra.run.smithi121.stdout:(37/155): cephadm-19.0.0-202.gc7586e95.el9.noar 7.9 MB/s | 805 kB 00:00 2023-12-16T21:11:23.869 INFO:teuthology.orchestra.run.smithi121.stdout:(38/155): cephfs-top-19.0.0-202.gc7586e95.el9.n 198 kB/s | 18 kB 00:00 2023-12-16T21:11:24.010 INFO:teuthology.orchestra.run.smithi121.stdout:(39/155): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2023-12-16T21:11:24.036 INFO:teuthology.orchestra.run.smithi121.stdout:(40/155): bison-3.7.4-5.el9.x86_64.rpm 5.4 MB/s | 920 kB 00:00 2023-12-16T21:11:24.070 INFO:teuthology.orchestra.run.smithi121.stdout:(41/155): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2023-12-16T21:11:24.087 INFO:teuthology.orchestra.run.smithi121.stdout:(42/155): elfutils-libelf-devel-0.190-1.el9.x86 428 kB/s | 21 kB 00:00 2023-12-16T21:11:24.112 INFO:teuthology.orchestra.run.smithi121.stdout:(43/155): flex-2.6.4-9.el9.x86_64.rpm 7.1 MB/s | 307 kB 00:00 2023-12-16T21:11:24.129 INFO:teuthology.orchestra.run.smithi121.stdout:(44/155): flexiblas-3.0.4-8.el9.x86_64.rpm 764 kB/s | 32 kB 00:00 2023-12-16T21:11:24.521 INFO:teuthology.orchestra.run.smithi121.stdout:(45/155): ceph-test-19.0.0-202.gc7586e95.el9.x8 12 MB/s | 41 MB 00:03 2023-12-16T21:11:24.571 INFO:teuthology.orchestra.run.smithi121.stdout:(46/155): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.5 MB/s | 3.0 MB 00:00 2023-12-16T21:11:24.588 INFO:teuthology.orchestra.run.smithi121.stdout:(47/155): flexiblas-openblas-openmp-3.0.4-8.el9 37 kB/s | 17 kB 00:00 2023-12-16T21:11:24.613 INFO:teuthology.orchestra.run.smithi121.stdout:(48/155): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2023-12-16T21:11:24.630 INFO:teuthology.orchestra.run.smithi121.stdout:(49/155): libaio-devel-0.3.111-13.el9.x86_64.rp 243 kB/s | 10 kB 00:00 2023-12-16T21:11:24.655 INFO:teuthology.orchestra.run.smithi121.stdout:(50/155): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2023-12-16T21:11:24.672 INFO:teuthology.orchestra.run.smithi121.stdout:(51/155): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2023-12-16T21:11:24.698 INFO:teuthology.orchestra.run.smithi121.stdout:(52/155): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2023-12-16T21:11:24.723 INFO:teuthology.orchestra.run.smithi121.stdout:(53/155): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2023-12-16T21:11:24.740 INFO:teuthology.orchestra.run.smithi121.stdout:(54/155): libuuid-devel-2.37.4-15.el9.x86_64.rp 371 kB/s | 24 kB 00:00 2023-12-16T21:11:24.757 INFO:teuthology.orchestra.run.smithi121.stdout:(55/155): libzstd-devel-1.5.1-2.el9.x86_64.rpm 807 kB/s | 46 kB 00:00 2023-12-16T21:11:24.782 INFO:teuthology.orchestra.run.smithi121.stdout:(56/155): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2023-12-16T21:11:24.807 INFO:teuthology.orchestra.run.smithi121.stdout:(57/155): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2023-12-16T21:11:24.832 INFO:teuthology.orchestra.run.smithi121.stdout:(58/155): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2023-12-16T21:11:24.858 INFO:teuthology.orchestra.run.smithi121.stdout:(59/155): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2023-12-16T21:11:24.925 INFO:teuthology.orchestra.run.smithi121.stdout:(60/155): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2023-12-16T21:11:24.966 INFO:teuthology.orchestra.run.smithi121.stdout:(61/155): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2023-12-16T21:11:25.000 INFO:teuthology.orchestra.run.smithi121.stdout:(62/155): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2023-12-16T21:11:25.035 INFO:teuthology.orchestra.run.smithi121.stdout:(63/155): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2023-12-16T21:11:25.060 INFO:teuthology.orchestra.run.smithi121.stdout:(64/155): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2023-12-16T21:11:25.152 INFO:teuthology.orchestra.run.smithi121.stdout:(65/155): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2023-12-16T21:11:25.236 INFO:teuthology.orchestra.run.smithi121.stdout:(66/155): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2023-12-16T21:11:25.261 INFO:teuthology.orchestra.run.smithi121.stdout:(67/155): python3-jinja2-2.11.3-4.el9.noarch.rp 1.1 MB/s | 228 kB 00:00 2023-12-16T21:11:25.286 INFO:teuthology.orchestra.run.smithi121.stdout:(68/155): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2023-12-16T21:11:25.303 INFO:teuthology.orchestra.run.smithi121.stdout:(69/155): python3-jsonpointer-2.0-4.el9.noarch. 272 kB/s | 18 kB 00:00 2023-12-16T21:11:25.320 INFO:teuthology.orchestra.run.smithi121.stdout:(70/155): python3-markupsafe-1.1.1-12.el9.x86_6 950 kB/s | 32 kB 00:00 2023-12-16T21:11:25.345 INFO:teuthology.orchestra.run.smithi121.stdout:(71/155): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2023-12-16T21:11:25.412 INFO:teuthology.orchestra.run.smithi121.stdout:(72/155): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 4.8 MB 00:00 2023-12-16T21:11:25.437 INFO:teuthology.orchestra.run.smithi121.stdout:(73/155): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2023-12-16T21:11:25.462 INFO:teuthology.orchestra.run.smithi121.stdout:(74/155): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 174 kB 00:00 2023-12-16T21:11:25.479 INFO:teuthology.orchestra.run.smithi121.stdout:(75/155): python3-prettytable-0.7.2-27.el9.noar 991 kB/s | 41 kB 00:00 2023-12-16T21:11:25.505 INFO:teuthology.orchestra.run.smithi121.stdout:(76/155): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-16T21:11:25.530 INFO:teuthology.orchestra.run.smithi121.stdout:(77/155): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2023-12-16T21:11:25.547 INFO:teuthology.orchestra.run.smithi121.stdout:(78/155): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2023-12-16T21:11:25.564 INFO:teuthology.orchestra.run.smithi121.stdout:(79/155): python3-requests-oauthlib-1.3.0-12.el 738 kB/s | 43 kB 00:00 2023-12-16T21:11:25.789 INFO:teuthology.orchestra.run.smithi121.stdout:(80/155): python3-scipy-1.6.2-8.el9.x86_64.rpm 61 MB/s | 16 MB 00:00 2023-12-16T21:11:25.806 INFO:teuthology.orchestra.run.smithi121.stdout:(81/155): python3-wcwidth-0.2.5-8.el9.noarch.rp 172 kB/s | 41 kB 00:00 2023-12-16T21:11:25.822 INFO:teuthology.orchestra.run.smithi121.stdout:(82/155): python3-toml-0.10.2-6.el9.noarch.rpm 151 kB/s | 39 kB 00:00 2023-12-16T21:11:25.839 INFO:teuthology.orchestra.run.smithi121.stdout:(83/155): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2023-12-16T21:11:25.865 INFO:teuthology.orchestra.run.smithi121.stdout:(84/155): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2023-12-16T21:11:25.890 INFO:teuthology.orchestra.run.smithi121.stdout:(85/155): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.2 MB/s | 85 kB 00:00 2023-12-16T21:11:25.915 INFO:teuthology.orchestra.run.smithi121.stdout:(86/155): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2023-12-16T21:11:25.932 INFO:teuthology.orchestra.run.smithi121.stdout:(87/155): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2023-12-16T21:11:25.957 INFO:teuthology.orchestra.run.smithi121.stdout:(88/155): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2023-12-16T21:11:25.982 INFO:teuthology.orchestra.run.smithi121.stdout:(89/155): libgfortran-11.4.1-2.3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2023-12-16T21:11:26.008 INFO:teuthology.orchestra.run.smithi121.stdout:(90/155): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2023-12-16T21:11:26.033 INFO:teuthology.orchestra.run.smithi121.stdout:(91/155): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-16T21:11:26.067 INFO:teuthology.orchestra.run.smithi121.stdout:(92/155): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2023-12-16T21:11:26.092 INFO:teuthology.orchestra.run.smithi121.stdout:(93/155): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-12-16T21:11:26.117 INFO:teuthology.orchestra.run.smithi121.stdout:(94/155): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2023-12-16T21:11:26.151 INFO:teuthology.orchestra.run.smithi121.stdout:(95/155): python3-requests-2.25.1-7.el9.noarch. 1.3 MB/s | 113 kB 00:00 2023-12-16T21:11:26.176 INFO:teuthology.orchestra.run.smithi121.stdout:(96/155): python3-urllib3-1.26.5-4.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2023-12-16T21:11:26.443 INFO:teuthology.orchestra.run.smithi121.stdout:(97/155): python3-devel-3.9.17-2.el9.x86_64.rpm 843 kB/s | 245 kB 00:00 2023-12-16T21:11:26.835 INFO:teuthology.orchestra.run.smithi121.stdout:(98/155): gperftools-libs-2.9.1-2.el9.x86_64.rp 469 kB/s | 309 kB 00:00 2023-12-16T21:11:26.943 INFO:teuthology.orchestra.run.smithi121.stdout:(99/155): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2023-12-16T21:11:27.060 INFO:teuthology.orchestra.run.smithi121.stdout:(100/155): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2023-12-16T21:11:27.186 INFO:teuthology.orchestra.run.smithi121.stdout:(101/155): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2023-12-16T21:11:27.319 INFO:teuthology.orchestra.run.smithi121.stdout:(102/155): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2023-12-16T21:11:27.461 INFO:teuthology.orchestra.run.smithi121.stdout:(103/155): openssl-devel-3.0.7-25.el9.x86_64.rp 3.1 MB/s | 4.1 MB 00:01 2023-12-16T21:11:27.594 INFO:teuthology.orchestra.run.smithi121.stdout:(104/155): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2023-12-16T21:11:27.653 INFO:teuthology.orchestra.run.smithi121.stdout:(105/155): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2023-12-16T21:11:27.728 INFO:teuthology.orchestra.run.smithi121.stdout:(106/155): python3-bcrypt-3.2.2-1.el9.x86_64.rp 404 kB/s | 43 kB 00:00 2023-12-16T21:11:27.762 INFO:teuthology.orchestra.run.smithi121.stdout:(107/155): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2023-12-16T21:11:27.870 INFO:teuthology.orchestra.run.smithi121.stdout:(108/155): python3-certifi-2021.10.8-2.el9.noar 105 kB/s | 15 kB 00:00 2023-12-16T21:11:27.913 INFO:teuthology.orchestra.run.smithi121.stdout:(109/155): python3-cheroot-8.6.0-4.el9.noarch.r 1.1 MB/s | 172 kB 00:00 2023-12-16T21:11:28.004 INFO:teuthology.orchestra.run.smithi121.stdout:(110/155): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2023-12-16T21:11:28.080 INFO:teuthology.orchestra.run.smithi121.stdout:(111/155): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2023-12-16T21:11:28.155 INFO:teuthology.orchestra.run.smithi121.stdout:(112/155): python3-colorama-0.4.4-12.el9.noarch 235 kB/s | 35 kB 00:00 2023-12-16T21:11:28.197 INFO:teuthology.orchestra.run.smithi121.stdout:(113/155): python3-google-auth-2.23.4-1.el9.noa 1.7 MB/s | 197 kB 00:00 2023-12-16T21:11:28.257 INFO:teuthology.orchestra.run.smithi121.stdout:(114/155): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2023-12-16T21:11:28.306 INFO:teuthology.orchestra.run.smithi121.stdout:(115/155): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2023-12-16T21:11:28.364 INFO:teuthology.orchestra.run.smithi121.stdout:(116/155): python3-asyncssh-2.13.2-3.el9.noarch 606 kB/s | 548 kB 00:00 2023-12-16T21:11:28.381 INFO:teuthology.orchestra.run.smithi121.stdout:(117/155): python3-jaraco-collections-3.0.0-8.e 184 kB/s | 23 kB 00:00 2023-12-16T21:11:28.448 INFO:teuthology.orchestra.run.smithi121.stdout:(118/155): python3-jaraco-functools-3.5.0-2.el9 137 kB/s | 19 kB 00:00 2023-12-16T21:11:28.490 INFO:teuthology.orchestra.run.smithi121.stdout:(119/155): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2023-12-16T21:11:28.507 INFO:teuthology.orchestra.run.smithi121.stdout:(120/155): python3-jwt+crypto-2.4.0-1.el9.noarc 72 kB/s | 9.0 kB 00:00 2023-12-16T21:11:28.549 INFO:teuthology.orchestra.run.smithi121.stdout:(121/155): python3-jwt-2.4.0-1.el9.noarch.rpm 406 kB/s | 41 kB 00:00 2023-12-16T21:11:28.716 INFO:teuthology.orchestra.run.smithi121.stdout:(122/155): python3-kubernetes-26.1.0-2.el9.noar 4.5 MB/s | 1.0 MB 00:00 2023-12-16T21:11:28.733 INFO:teuthology.orchestra.run.smithi121.stdout:(123/155): python3-logutils-0.3.5-21.el9.noarch 205 kB/s | 46 kB 00:00 2023-12-16T21:11:28.750 INFO:teuthology.orchestra.run.smithi121.stdout:(124/155): python3-more-itertools-8.12.0-2.el9. 393 kB/s | 79 kB 00:00 2023-12-16T21:11:28.859 INFO:teuthology.orchestra.run.smithi121.stdout:(125/155): python3-natsort-7.1.1-5.el9.noarch.r 406 kB/s | 58 kB 00:00 2023-12-16T21:11:28.884 INFO:teuthology.orchestra.run.smithi121.stdout:(126/155): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2023-12-16T21:11:28.917 INFO:teuthology.orchestra.run.smithi121.stdout:(127/155): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2023-12-16T21:11:28.976 INFO:teuthology.orchestra.run.smithi121.stdout:(128/155): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2023-12-16T21:11:29.001 INFO:teuthology.orchestra.run.smithi121.stdout:(129/155): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2023-12-16T21:11:29.018 INFO:teuthology.orchestra.run.smithi121.stdout:(130/155): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2023-12-16T21:11:29.094 INFO:teuthology.orchestra.run.smithi121.stdout:(131/155): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2023-12-16T21:11:29.119 INFO:teuthology.orchestra.run.smithi121.stdout:(132/155): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2023-12-16T21:11:29.144 INFO:teuthology.orchestra.run.smithi121.stdout:(133/155): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2023-12-16T21:11:29.203 INFO:teuthology.orchestra.run.smithi121.stdout:(134/155): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2023-12-16T21:11:29.261 INFO:teuthology.orchestra.run.smithi121.stdout:(135/155): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2023-12-16T21:11:29.286 INFO:teuthology.orchestra.run.smithi121.stdout:(136/155): python3-websocket-client-1.2.3-2.el9 631 kB/s | 90 kB 00:00 2023-12-16T21:11:29.328 INFO:teuthology.orchestra.run.smithi121.stdout:(137/155): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2023-12-16T21:11:29.370 INFO:teuthology.orchestra.run.smithi121.stdout:(138/155): python3-zc-lockfile-2.0-10.el9.noarc 183 kB/s | 20 kB 00:00 2023-12-16T21:11:29.423 INFO:teuthology.orchestra.run.smithi121.stdout:(139/155): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2023-12-16T21:11:29.506 INFO:teuthology.orchestra.run.smithi121.stdout:(140/155): thrift-0.15.0-2.el9.x86_64.rpm 8.9 MB/s | 1.6 MB 00:00 2023-12-16T21:11:29.540 INFO:teuthology.orchestra.run.smithi121.stdout:(141/155): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2023-12-16T21:11:29.574 INFO:teuthology.orchestra.run.smithi121.stdout:(142/155): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2023-12-16T21:11:29.600 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2023-12-16T21:11:29.607 INFO:teuthology.orchestra.run.smithi121.stdout:(143/155): elfutils-debuginfod-client-0.190-1.e 1.1 MB/s | 36 kB 00:00 2023-12-16T21:11:29.641 INFO:teuthology.orchestra.run.smithi121.stdout:(144/155): elfutils-libelf-0.190-1.el9.x86_64.r 5.7 MB/s | 192 kB 00:00 2023-12-16T21:11:29.674 INFO:teuthology.orchestra.run.smithi121.stdout:(145/155): elfutils-libs-0.190-1.el9.x86_64.rpm 7.5 MB/s | 255 kB 00:00 2023-12-16T21:11:29.716 INFO:teuthology.orchestra.run.smithi121.stdout:(146/155): libacl-2.3.1-4.el9.x86_64.rpm 527 kB/s | 22 kB 00:00 2023-12-16T21:11:29.723 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/170 2023-12-16T21:11:29.750 INFO:teuthology.orchestra.run.smithi121.stdout:(147/155): xfsdump-3.1.12-4.el9.x86_64.rpm 9.4 MB/s | 321 kB 00:00 2023-12-16T21:11:29.756 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/170 2023-12-16T21:11:29.798 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/170 2023-12-16T21:11:29.839 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/170 2023-12-16T21:11:29.865 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2023-12-16T21:11:29.915 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2023-12-16T21:11:29.950 INFO:teuthology.orchestra.run.smithi121.stdout:(148/155): librados2-19.0.0-202.gc7586e95.el9.x 5.7 MB/s | 3.3 MB 00:00 2023-12-16T21:11:30.000 INFO:teuthology.orchestra.run.smithi121.stdout:(149/155): librbd1-19.0.0-202.gc7586e95.el9.x86 5.2 MB/s | 3.0 MB 00:00 2023-12-16T21:11:30.134 INFO:teuthology.orchestra.run.smithi121.stdout:(150/155): NetworkManager-1.45.7-1.el9.x86_64.r 5.9 MB/s | 2.3 MB 00:00 2023-12-16T21:11:30.152 INFO:teuthology.orchestra.run.smithi121.stdout:(151/155): NetworkManager-team-1.45.7-1.el9.x86 287 kB/s | 43 kB 00:00 2023-12-16T21:11:30.193 INFO:teuthology.orchestra.run.smithi121.stdout:(152/155): NetworkManager-tui-1.45.7-1.el9.x86_ 4.1 MB/s | 246 kB 00:00 2023-12-16T21:11:30.254 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2023-12-16T21:11:30.294 INFO:teuthology.orchestra.run.smithi121.stdout:(153/155): openssl-libs-3.0.7-25.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2023-12-16T21:11:30.336 INFO:teuthology.orchestra.run.smithi121.stdout:(154/155): NetworkManager-libnm-1.45.7-1.el9.x8 4.6 MB/s | 1.8 MB 00:00 2023-12-16T21:11:30.369 INFO:teuthology.orchestra.run.smithi121.stdout:(155/155): openssl-3.0.7-25.el9.x86_64.rpm 5.6 MB/s | 1.2 MB 00:00 2023-12-16T21:11:30.377 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:11:30.377 INFO:teuthology.orchestra.run.smithi121.stdout:Total 17 MB/s | 205 MB 00:11 2023-12-16T21:11:30.382 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-202.gc7586e95.el9.x86_64 8/170 2023-12-16T21:11:30.540 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-12-16T21:11:30.589 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-202.gc7586e95.el9.x86_64 8/170 2023-12-16T21:11:30.625 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 9/170 2023-12-16T21:11:30.718 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 9/170 2023-12-16T21:11:30.729 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-12-16T21:11:30.729 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-12-16T21:11:30.750 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-202.gc7586e95.el9.x86_64 10/170 2023-12-16T21:11:30.792 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/170 2023-12-16T21:11:30.989 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/170 2023-12-16T21:11:31.044 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/170 2023-12-16T21:11:31.071 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 14/170 2023-12-16T21:11:31.166 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 14/170 2023-12-16T21:11:31.189 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_ 15/170 2023-12-16T21:11:31.409 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_ 15/170 2023-12-16T21:11:31.537 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-16T21:11:31.580 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-16T21:11:31.675 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-16T21:11:31.723 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-16T21:11:31.922 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-16T21:11:32.026 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 18/170 2023-12-16T21:11:32.060 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/170 2023-12-16T21:11:32.140 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/170 2023-12-16T21:11:32.193 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/170 2023-12-16T21:11:32.222 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/170 2023-12-16T21:11:32.258 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/170 2023-12-16T21:11:32.307 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/170 2023-12-16T21:11:32.391 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/170 2023-12-16T21:11:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/170 2023-12-16T21:11:32.458 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/170 2023-12-16T21:11:32.530 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2023-12-16T21:11:32.550 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-12-16T21:11:32.550 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-12-16T21:11:32.563 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-202.gc7586e95.el9.x 29/170 2023-12-16T21:11:32.607 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-202.gc7586e95.el9 30/170 2023-12-16T21:11:32.691 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-202.gc7586e95.el9.x86_64 31/170 2023-12-16T21:11:32.828 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 32/170 2023-12-16T21:11:32.885 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 33/170 2023-12-16T21:11:33.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 34/170 2023-12-16T21:11:33.540 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 35/170 2023-12-16T21:11:33.594 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 36/170 2023-12-16T21:11:33.644 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 37/170 2023-12-16T21:11:33.681 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-202.gc7586e95.el9.x86_64 38/170 2023-12-16T21:11:33.724 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 39/170 2023-12-16T21:11:33.881 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 40/170 2023-12-16T21:11:33.923 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 41/170 2023-12-16T21:11:34.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 42/170 2023-12-16T21:11:34.068 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 43/170 2023-12-16T21:11:34.124 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/170 2023-12-16T21:11:34.198 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/170 2023-12-16T21:11:34.236 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/170 2023-12-16T21:11:34.272 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/170 2023-12-16T21:11:34.320 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/170 2023-12-16T21:11:34.355 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/170 2023-12-16T21:11:34.398 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/170 2023-12-16T21:11:34.469 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/170 2023-12-16T21:11:34.604 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/170 2023-12-16T21:11:34.654 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/170 2023-12-16T21:11:34.684 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/170 2023-12-16T21:11:34.726 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/170 2023-12-16T21:11:34.822 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/170 2023-12-16T21:11:34.860 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 57/170 2023-12-16T21:11:34.909 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/170 2023-12-16T21:11:34.941 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/170 2023-12-16T21:11:35.039 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/170 2023-12-16T21:11:35.094 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 61/170 2023-12-16T21:11:35.152 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 62/170 2023-12-16T21:11:35.216 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 63/170 2023-12-16T21:11:35.413 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 64/170 2023-12-16T21:11:35.457 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 65/170 2023-12-16T21:11:35.599 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 66/170 2023-12-16T21:11:35.643 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 67/170 2023-12-16T21:11:35.680 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 68/170 2023-12-16T21:11:35.714 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 69/170 2023-12-16T21:11:35.748 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 70/170 2023-12-16T21:11:35.778 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 71/170 2023-12-16T21:11:35.806 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 72/170 2023-12-16T21:11:35.850 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/170 2023-12-16T21:11:35.888 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/170 2023-12-16T21:11:36.003 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/170 2023-12-16T21:11:36.033 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/170 2023-12-16T21:11:36.114 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/170 2023-12-16T21:11:36.152 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/170 2023-12-16T21:11:36.185 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/170 2023-12-16T21:11:36.227 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/170 2023-12-16T21:11:36.262 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/170 2023-12-16T21:11:36.355 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/170 2023-12-16T21:11:36.389 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/170 2023-12-16T21:11:36.495 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/170 2023-12-16T21:11:36.549 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2023-12-16T21:11:37.314 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/170 2023-12-16T21:11:37.353 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/170 2023-12-16T21:11:37.387 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/170 2023-12-16T21:11:37.436 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/170 2023-12-16T21:11:37.526 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/170 2023-12-16T21:11:37.555 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/170 2023-12-16T21:11:37.730 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 92/170 2023-12-16T21:11:37.769 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 93/170 2023-12-16T21:11:37.848 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/170 2023-12-16T21:11:38.532 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/170 2023-12-16T21:11:38.649 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-12-16T21:11:38.652 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/170 2023-12-16T21:11:38.779 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/170 2023-12-16T21:11:38.806 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/170 2023-12-16T21:11:38.850 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/170 2023-12-16T21:11:38.888 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/170 2023-12-16T21:11:38.914 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2023-12-16T21:11:38.965 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2023-12-16T21:11:39.301 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2023-12-16T21:11:39.349 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-202.gc7586e95.el9.x86_64 8/170 2023-12-16T21:11:39.451 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-202.gc7586e95.el9.x86_64 8/170 2023-12-16T21:11:39.475 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 9/170 2023-12-16T21:11:39.566 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 9/170 2023-12-16T21:11:39.600 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-202.gc7586e95.el9.x86_64 10/170 2023-12-16T21:11:39.640 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/170 2023-12-16T21:11:39.822 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/170 2023-12-16T21:11:39.869 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/170 2023-12-16T21:11:39.897 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 14/170 2023-12-16T21:11:39.946 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/170 2023-12-16T21:11:39.997 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 14/170 2023-12-16T21:11:40.023 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_ 15/170 2023-12-16T21:11:40.028 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/170 2023-12-16T21:11:40.068 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 99/170 2023-12-16T21:11:40.105 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 100/170 2023-12-16T21:11:40.179 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 101/170 2023-12-16T21:11:40.217 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 102/170 2023-12-16T21:11:40.237 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_ 15/170 2023-12-16T21:11:40.246 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 103/170 2023-12-16T21:11:40.279 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 104/170 2023-12-16T21:11:40.368 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-16T21:11:40.429 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-16T21:11:40.515 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-16T21:11:40.564 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-16T21:11:40.595 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 105/170 2023-12-16T21:11:40.625 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 106/170 2023-12-16T21:11:40.717 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 106/170 2023-12-16T21:11:40.769 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-16T21:11:40.772 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-202.gc7586e95.el9.x86_64 107/170 2023-12-16T21:11:40.808 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 108/170 2023-12-16T21:11:40.874 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 18/170 2023-12-16T21:11:40.910 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/170 2023-12-16T21:11:40.980 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/170 2023-12-16T21:11:41.032 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/170 2023-12-16T21:11:41.063 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/170 2023-12-16T21:11:41.107 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/170 2023-12-16T21:11:41.147 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/170 2023-12-16T21:11:41.212 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/170 2023-12-16T21:11:41.254 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/170 2023-12-16T21:11:41.257 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 109/170 2023-12-16T21:11:41.282 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/170 2023-12-16T21:11:41.286 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 110/170 2023-12-16T21:11:41.352 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2023-12-16T21:11:41.357 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 110/170 2023-12-16T21:11:41.387 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-202.gc7586e95.el9.x 29/170 2023-12-16T21:11:41.388 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-202.gc7586e95.el9.x86_64 111/170 2023-12-16T21:11:41.434 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-202.gc7586e95.el9 30/170 2023-12-16T21:11:41.512 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-202.gc7586e95.el9.x86_64 31/170 2023-12-16T21:11:41.646 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 32/170 2023-12-16T21:11:41.699 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 33/170 2023-12-16T21:11:42.216 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 34/170 2023-12-16T21:11:42.293 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 35/170 2023-12-16T21:11:42.342 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 36/170 2023-12-16T21:11:42.387 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 37/170 2023-12-16T21:11:42.421 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-202.gc7586e95.el9.x86_64 38/170 2023-12-16T21:11:42.463 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 39/170 2023-12-16T21:11:42.601 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 40/170 2023-12-16T21:11:42.663 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 41/170 2023-12-16T21:11:42.769 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 42/170 2023-12-16T21:11:42.800 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 43/170 2023-12-16T21:11:42.854 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/170 2023-12-16T21:11:42.927 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/170 2023-12-16T21:11:42.959 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/170 2023-12-16T21:11:42.995 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/170 2023-12-16T21:11:43.042 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/170 2023-12-16T21:11:43.078 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/170 2023-12-16T21:11:43.121 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/170 2023-12-16T21:11:43.163 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/170 2023-12-16T21:11:43.294 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/170 2023-12-16T21:11:43.336 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/170 2023-12-16T21:11:43.366 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/170 2023-12-16T21:11:43.407 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/170 2023-12-16T21:11:43.499 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/170 2023-12-16T21:11:43.545 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 57/170 2023-12-16T21:11:43.590 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/170 2023-12-16T21:11:43.623 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/170 2023-12-16T21:11:43.628 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 112/170 2023-12-16T21:11:43.668 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 112/170 2023-12-16T21:11:43.716 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/170 2023-12-16T21:11:43.767 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 61/170 2023-12-16T21:11:43.816 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 62/170 2023-12-16T21:11:43.871 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 63/170 2023-12-16T21:11:44.062 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 64/170 2023-12-16T21:11:44.113 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 65/170 2023-12-16T21:11:44.237 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 112/170 2023-12-16T21:11:44.239 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 66/170 2023-12-16T21:11:44.272 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 113/170 2023-12-16T21:11:44.282 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 67/170 2023-12-16T21:11:44.312 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 68/170 2023-12-16T21:11:44.348 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 69/170 2023-12-16T21:11:44.364 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 113/170 2023-12-16T21:11:44.364 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T21:11:44.365 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T21:11:44.365 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:44.389 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 70/170 2023-12-16T21:11:44.406 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 114/170 2023-12-16T21:11:44.422 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 71/170 2023-12-16T21:11:44.448 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 72/170 2023-12-16T21:11:44.483 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/170 2023-12-16T21:11:44.521 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/170 2023-12-16T21:11:44.552 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/170 2023-12-16T21:11:44.583 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/170 2023-12-16T21:11:44.658 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/170 2023-12-16T21:11:44.693 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/170 2023-12-16T21:11:44.726 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/170 2023-12-16T21:11:44.768 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/170 2023-12-16T21:11:44.802 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/170 2023-12-16T21:11:44.890 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/170 2023-12-16T21:11:44.947 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/170 2023-12-16T21:11:45.048 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/170 2023-12-16T21:11:45.097 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2023-12-16T21:11:45.813 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/170 2023-12-16T21:11:45.861 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/170 2023-12-16T21:11:45.895 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/170 2023-12-16T21:11:45.943 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/170 2023-12-16T21:11:46.022 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/170 2023-12-16T21:11:46.055 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/170 2023-12-16T21:11:46.234 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 92/170 2023-12-16T21:11:46.261 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 93/170 2023-12-16T21:11:46.339 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/170 2023-12-16T21:11:47.006 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/170 2023-12-16T21:11:47.115 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/170 2023-12-16T21:11:48.330 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/170 2023-12-16T21:11:48.412 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/170 2023-12-16T21:11:48.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 99/170 2023-12-16T21:11:48.480 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-16.el9.x86_64 100/170 2023-12-16T21:11:48.552 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 101/170 2023-12-16T21:11:48.584 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 102/170 2023-12-16T21:11:48.655 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 103/170 2023-12-16T21:11:48.688 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 104/170 2023-12-16T21:11:48.986 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 105/170 2023-12-16T21:11:49.018 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 106/170 2023-12-16T21:11:49.284 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 106/170 2023-12-16T21:11:49.314 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-202.gc7586e95.el9.x86_64 107/170 2023-12-16T21:11:49.351 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 108/170 2023-12-16T21:11:49.780 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 109/170 2023-12-16T21:11:49.812 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 110/170 2023-12-16T21:11:49.883 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 110/170 2023-12-16T21:11:49.905 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-202.gc7586e95.el9.x86_64 111/170 2023-12-16T21:11:51.943 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 112/170 2023-12-16T21:11:51.997 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 112/170 2023-12-16T21:11:52.534 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 112/170 2023-12-16T21:11:52.581 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 113/170 2023-12-16T21:11:52.661 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 113/170 2023-12-16T21:11:52.661 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T21:11:52.661 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T21:11:52.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:11:52.699 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 114/170 2023-12-16T21:11:56.242 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 114/170 2023-12-16T21:11:56.242 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2023-12-16T21:11:56.242 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2023-12-16T21:11:56.242 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2023-12-16T21:11:56.242 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2023-12-16T21:11:56.242 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2023-12-16T21:11:56.243 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2023-12-16T21:11:56.243 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2023-12-16T21:11:56.243 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:56.411 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 115/170 2023-12-16T21:11:56.427 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 115/170 2023-12-16T21:11:56.427 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:11:56.428 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-16T21:11:56.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:58.057 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 116/170 2023-12-16T21:11:58.084 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 116/170 2023-12-16T21:11:58.084 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:11:58.084 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T21:11:58.084 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:11:58.084 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:11:58.084 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:58.299 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 117/170 2023-12-16T21:11:58.323 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 117/170 2023-12-16T21:11:58.323 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:11:58.323 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T21:11:58.323 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:11:58.323 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:11:58.324 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:58.791 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 118/170 2023-12-16T21:11:58.815 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 118/170 2023-12-16T21:11:58.816 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:11:58.816 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T21:11:58.816 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:11:58.816 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:11:58.816 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:11:58.910 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 119/170 2023-12-16T21:11:59.324 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-202.gc7586e95.el9.noarch 120/170 2023-12-16T21:11:59.358 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-202.gc7586e95.el9.noarch 120/170 2023-12-16T21:11:59.389 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-202.gc7586e95.el 121/170 2023-12-16T21:11:59.452 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-202.gc7586e95.e 122/170 2023-12-16T21:11:59.490 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noar 123/170 2023-12-16T21:11:59.727 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noar 123/170 2023-12-16T21:11:59.766 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.no 124/170 2023-12-16T21:12:00.781 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.no 124/170 2023-12-16T21:12:00.808 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-202.gc758 125/170 2023-12-16T21:12:00.912 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-202.gc758 125/170 2023-12-16T21:12:01.072 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-202.gc7586e95.el9 126/170 2023-12-16T21:12:01.101 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 127/170 2023-12-16T21:12:01.126 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 127/170 2023-12-16T21:12:01.126 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:01.126 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T21:12:01.126 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:12:01.126 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:12:01.126 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:01.177 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 128/170 2023-12-16T21:12:01.188 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 128/170 2023-12-16T21:12:01.237 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-202.gc7586e95.el9.x86_64 129/170 2023-12-16T21:12:02.053 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/170 2023-12-16T21:12:02.087 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 131/170 2023-12-16T21:12:02.111 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 131/170 2023-12-16T21:12:02.112 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:02.112 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T21:12:02.112 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:12:02.112 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:12:02.112 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:02.155 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-202.gc7586e 132/170 2023-12-16T21:12:02.179 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-202.gc7586e 132/170 2023-12-16T21:12:02.179 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:02.179 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T21:12:02.179 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:02.239 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 133/170 2023-12-16T21:12:02.263 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 133/170 2023-12-16T21:12:02.263 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:02.263 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-12-16T21:12:02.263 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:12:02.263 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:12:02.263 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:02.581 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 134/170 2023-12-16T21:12:02.605 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 134/170 2023-12-16T21:12:02.606 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:02.606 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T21:12:02.606 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:12:02.606 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:12:02.606 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 114/170 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2023-12-16T21:12:04.206 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:04.368 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 115/170 2023-12-16T21:12:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 115/170 2023-12-16T21:12:04.384 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:04.384 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-16T21:12:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:05.974 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 116/170 2023-12-16T21:12:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 116/170 2023-12-16T21:12:05.999 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:05.999 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T21:12:05.999 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:12:05.999 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:12:05.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:06.199 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 117/170 2023-12-16T21:12:06.221 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 117/170 2023-12-16T21:12:06.221 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:06.221 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T21:12:06.221 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:12:06.221 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:12:06.221 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:06.652 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-202.gc7586e95.el9.x86_64 135/170 2023-12-16T21:12:06.667 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 118/170 2023-12-16T21:12:06.689 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 118/170 2023-12-16T21:12:06.689 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:06.689 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T21:12:06.690 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:12:06.690 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:12:06.690 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:06.693 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 136/170 2023-12-16T21:12:06.727 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-202.gc7586e95.el9.x86_64 137/170 2023-12-16T21:12:06.749 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 138/170 2023-12-16T21:12:06.777 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 139/170 2023-12-16T21:12:06.785 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.2-8.el9.noarch 119/170 2023-12-16T21:12:06.978 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-202.gc7586e95.el9.x86_64 140/170 2023-12-16T21:12:07.094 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 141/170 2023-12-16T21:12:07.123 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 142/170 2023-12-16T21:12:07.164 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-202.gc7586e95.el9.x86_6 143/170 2023-12-16T21:12:07.173 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-202.gc7586e95.el9.noarch 120/170 2023-12-16T21:12:07.221 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 144/170 2023-12-16T21:12:07.225 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-202.gc7586e95.el9.noarch 120/170 2023-12-16T21:12:07.271 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-202.gc7586e95.el 121/170 2023-12-16T21:12:07.272 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-16T21:12:07.308 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 146/170 2023-12-16T21:12:07.332 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-202.gc7586e95.e 122/170 2023-12-16T21:12:07.337 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-team-1:1.45.7-1.el9.x86_64 147/170 2023-12-16T21:12:07.365 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noar 123/170 2023-12-16T21:12:07.383 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-tui-1:1.45.7-1.el9.x86_64 148/170 2023-12-16T21:12:07.591 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noar 123/170 2023-12-16T21:12:07.624 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.no 124/170 2023-12-16T21:12:07.916 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-16T21:12:07.948 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-16T21:12:07.969 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-16T21:12:08.590 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.no 124/170 2023-12-16T21:12:08.624 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-202.gc758 125/170 2023-12-16T21:12:08.723 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-202.gc758 125/170 2023-12-16T21:12:08.842 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-202.gc7586e95.el9.noarch 150/170 2023-12-16T21:12:08.873 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-202.gc7586e95.el9 126/170 2023-12-16T21:12:08.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 127/170 2023-12-16T21:12:08.932 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 127/170 2023-12-16T21:12:08.932 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:08.932 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T21:12:08.932 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:12:08.932 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:12:08.932 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:08.961 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 151/170 2023-12-16T21:12:08.977 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 128/170 2023-12-16T21:12:08.987 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 128/170 2023-12-16T21:12:09.011 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 152/170 2023-12-16T21:12:09.027 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-202.gc7586e95.el9.x86_64 129/170 2023-12-16T21:12:09.054 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 153/170 2023-12-16T21:12:09.162 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 154/170 2023-12-16T21:12:09.198 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 155/170 2023-12-16T21:12:09.223 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 156/170 2023-12-16T21:12:09.249 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 157/170 2023-12-16T21:12:09.275 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 158/170 2023-12-16T21:12:09.275 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-16T21:12:09.360 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-16T21:12:09.388 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 160/170 2023-12-16T21:12:09.426 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/170 2023-12-16T21:12:09.451 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 162/170 2023-12-16T21:12:09.452 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-16T21:12:09.519 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-16T21:12:09.544 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-16T21:12:09.551 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-16T21:12:09.551 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-16T21:12:09.738 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-16T21:12:09.739 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-16T21:12:09.798 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/170 2023-12-16T21:12:09.823 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-16T21:12:09.828 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 131/170 2023-12-16T21:12:09.851 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 131/170 2023-12-16T21:12:09.851 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:09.851 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T21:12:09.851 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:12:09.852 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:12:09.852 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:09.857 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 167/170 2023-12-16T21:12:09.882 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 168/170 2023-12-16T21:12:09.896 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-202.gc7586e 132/170 2023-12-16T21:12:09.908 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 169/170 2023-12-16T21:12:09.918 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-202.gc7586e 132/170 2023-12-16T21:12:09.918 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:09.918 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T21:12:09.918 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:09.972 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 133/170 2023-12-16T21:12:09.975 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-16T21:12:09.994 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 133/170 2023-12-16T21:12:09.994 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:09.994 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-12-16T21:12:09.994 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:12:09.994 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-16T21:12:09.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:10.289 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 134/170 2023-12-16T21:12:10.312 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 134/170 2023-12-16T21:12:10.312 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:12:10.312 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T21:12:10.312 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:12:10.312 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:12:10.312 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:14.134 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-202.gc7586e95.el9.x86_64 135/170 2023-12-16T21:12:14.191 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 136/170 2023-12-16T21:12:14.232 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-202.gc7586e95.el9.x86_64 137/170 2023-12-16T21:12:14.265 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 138/170 2023-12-16T21:12:14.292 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 139/170 2023-12-16T21:12:14.401 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-16T21:12:14.401 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-202.gc7586e95.el9.x86_64 1/170 2023-12-16T21:12:14.401 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 2/170 2023-12-16T21:12:14.401 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 3/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 4/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-202.gc7586e 5/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 6/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 7/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 8/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 9/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 10/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 11/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-202.gc7586e95.el9.x86_64 12/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 13/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-202.gc7586e95.el9.x86_64 14/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-202.gc7586e95.el9.x86_6 15/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 16/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 17/170 2023-12-16T21:12:14.402 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-202.gc7586e95.el9.x86_64 18/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_ 19/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 20/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-202.gc7586e95.el9 21/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-202.gc7586e95.el9.x 22/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-202.gc7586e95.el9.x86_64 23/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-202.gc7586e95.el9.x86_64 24/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-202.gc7586e95.el9.x86_64 25/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-202.gc7586e95.el9.x86_64 26/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 27/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 28/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-202.gc7586e95.el9.x86_64 29/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-202.gc7586e95.e 30/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noar 31/170 2023-12-16T21:12:14.403 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.no 32/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-202.gc758 33/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-202.gc7586e95.el9 34/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 35/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-202.gc7586e95.el 36/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 37/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-202.gc7586e95.el9.noarch 38/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-202.gc7586e95.el9.noarch 39/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 43/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/170 2023-12-16T21:12:14.404 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 48/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 49/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 53/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 54/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 55/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 56/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 57/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 58/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 59/170 2023-12-16T21:12:14.405 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 60/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.11-1.el9.x86_64 61/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 62/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 63/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 64/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 67/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/170 2023-12-16T21:12:14.406 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 75/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 76/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 77/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 78/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 80/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 81/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 82/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 83/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 84/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 85/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 86/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 87/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 88/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 89/170 2023-12-16T21:12:14.407 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 90/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 91/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 92/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 93/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 94/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 95/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 96/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 97/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 98/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 99/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 100/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 101/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 102/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 103/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 104/170 2023-12-16T21:12:14.408 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 105/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 106/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 107/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 108/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 109/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 110/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 111/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 112/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 113/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 114/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 115/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 116/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 117/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 118/170 2023-12-16T21:12:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 119/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 120/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 121/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 122/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 123/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 124/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 125/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 126/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 127/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 128/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 129/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 130/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 131/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 132/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 133/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 134/170 2023-12-16T21:12:14.410 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 135/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 136/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 137/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 138/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 139/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 140/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-202.gc7586e95.el9.x86_64 141/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 142/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 143/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 144/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 146/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 149/170 2023-12-16T21:12:14.411 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 151/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 153/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 157/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 158/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.45.7-1.el9.x86_64 159/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 160/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 161/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 162/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.45.7-1.el9.x86_64 163/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-16T21:12:14.412 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.45.7-1.el9.x86_64 165/170 2023-12-16T21:12:14.413 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 166/170 2023-12-16T21:12:14.413 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 167/170 2023-12-16T21:12:14.413 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/170 2023-12-16T21:12:14.413 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 169/170 2023-12-16T21:12:14.481 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-shell-2:19.0.0-202.gc7586e95.el9.x86_64 140/170 2023-12-16T21:12:14.598 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 141/170 2023-12-16T21:12:14.631 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 142/170 2023-12-16T21:12:14.685 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-202.gc7586e95.el9.x86_6 143/170 2023-12-16T21:12:14.753 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 144/170 2023-12-16T21:12:14.820 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-16T21:12:14.848 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 146/170 2023-12-16T21:12:14.878 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-team-1:1.45.7-1.el9.x86_64 147/170 2023-12-16T21:12:14.915 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-tui-1:1.45.7-1.el9.x86_64 148/170 2023-12-16T21:12:15.494 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-16T21:12:15.542 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-16T21:12:15.561 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-16T21:12:16.363 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:19.0.0-202.gc7586e95.el9.noarch 150/170 2023-12-16T21:12:16.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 151/170 2023-12-16T21:12:16.496 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 152/170 2023-12-16T21:12:16.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 153/170 2023-12-16T21:12:16.635 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.4-9.el9.x86_64 154/170 2023-12-16T21:12:16.675 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.7.4-5.el9.x86_64 155/170 2023-12-16T21:12:16.700 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 156/170 2023-12-16T21:12:16.725 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 157/170 2023-12-16T21:12:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 158/170 2023-12-16T21:12:16.752 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/170 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: acl-2.3.1-4.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-12-16T21:12:16.789 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-2.3.1-4.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.790 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.791 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-12-16T21:12:16.792 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.793 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.11-1.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-12-16T21:12:16.794 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.11-1.el9.x86_64 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2023-12-16T21:12:16.795 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T21:12:16.796 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T21:12:16.797 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T21:12:16.798 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-12-16T21:12:16.799 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T21:12:16.800 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-12-16T21:12:16.800 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:16.800 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2023-12-16T21:12:16.820 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-16T21:12:16.848 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 160/170 2023-12-16T21:12:16.878 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/170 2023-12-16T21:12:16.903 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 162/170 2023-12-16T21:12:16.903 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-16T21:12:16.980 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-16T21:12:17.004 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-16T21:12:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-16T21:12:17.011 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-16T21:12:17.191 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-16T21:12:17.191 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-16T21:12:17.268 DEBUG:teuthology.parallel:result is None 2023-12-16T21:12:17.275 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-16T21:12:17.299 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 167/170 2023-12-16T21:12:17.324 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 168/170 2023-12-16T21:12:17.350 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 169/170 2023-12-16T21:12:17.415 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-202.gc7586e95.el9.x86_64 1/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 2/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 3/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 4/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-202.gc7586e 5/170 2023-12-16T21:12:21.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 6/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 7/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 8/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 9/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 10/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 11/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-202.gc7586e95.el9.x86_64 12/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 13/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-shell-2:19.0.0-202.gc7586e95.el9.x86_64 14/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-202.gc7586e95.el9.x86_6 15/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 16/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 17/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-202.gc7586e95.el9.x86_64 18/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_ 19/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 20/170 2023-12-16T21:12:21.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-202.gc7586e95.el9 21/170 2023-12-16T21:12:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-202.gc7586e95.el9.x 22/170 2023-12-16T21:12:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-202.gc7586e95.el9.x86_64 23/170 2023-12-16T21:12:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-202.gc7586e95.el9.x86_64 24/170 2023-12-16T21:12:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-202.gc7586e95.el9.x86_64 25/170 2023-12-16T21:12:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-202.gc7586e95.el9.x86_64 26/170 2023-12-16T21:12:21.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 27/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 28/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-202.gc7586e95.el9.x86_64 29/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-202.gc7586e95.e 30/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noar 31/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.no 32/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-202.gc758 33/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-202.gc7586e95.el9 34/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 35/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-202.gc7586e95.el 36/170 2023-12-16T21:12:22.006 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 37/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-202.gc7586e95.el9.noarch 38/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:19.0.0-202.gc7586e95.el9.noarch 39/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 43/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-16.el9.x86_64 48/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 49/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/170 2023-12-16T21:12:22.007 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 53/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 54/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 55/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 56/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 57/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 58/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 59/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 60/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-13.11-1.el9.x86_64 61/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 62/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 63/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 64/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/170 2023-12-16T21:12:22.008 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 67/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 75/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 76/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 77/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 78/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 80/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 81/170 2023-12-16T21:12:22.009 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 82/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 83/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 84/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 85/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 86/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 87/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 88/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 89/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 90/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 91/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 92/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 93/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 94/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 95/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 96/170 2023-12-16T21:12:22.010 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 97/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 98/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 99/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 100/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 101/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 102/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 103/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 104/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 105/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 106/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 107/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 108/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 109/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 110/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 111/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 112/170 2023-12-16T21:12:22.011 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 113/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 114/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 115/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 116/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 117/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 118/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 119/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 120/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 121/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 122/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 123/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 124/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 125/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 126/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 127/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 128/170 2023-12-16T21:12:22.012 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 129/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 130/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 131/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 132/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 133/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 134/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 135/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 136/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 137/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 138/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 139/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 140/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-202.gc7586e95.el9.x86_64 141/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 142/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 143/170 2023-12-16T21:12:22.013 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 144/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 146/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 149/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 151/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 153/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 157/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 158/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-1:1.45.7-1.el9.x86_64 159/170 2023-12-16T21:12:22.014 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 160/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 161/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 162/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-team-1:1.45.7-1.el9.x86_64 163/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-tui-1:1.45.7-1.el9.x86_64 165/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 166/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 167/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/170 2023-12-16T21:12:22.015 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 169/170 2023-12-16T21:12:24.553 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/170 2023-12-16T21:12:24.553 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:24.553 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2023-12-16T21:12:24.553 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:24.553 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:24.553 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-team-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-tui-1:1.45.7-1.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: acl-2.3.1-4.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: libacl-2.3.1-4.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-12-16T21:12:24.554 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.2-8.el9.noarch 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.7.4-5.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.555 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:19.0.0-202.gc7586e95.el9.noarch 2023-12-16T21:12:24.556 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.4-9.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-16.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-16T21:12:24.557 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-45.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T21:12:24.558 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-13.11-1.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-13.11-1.el9.x86_64 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T21:12:24.559 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs-20.3.0-7.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2-2.4.2-2.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama-0.4.4-12.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T21:12:24.560 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T21:12:24.561 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T21:12:24.562 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.564 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.564 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T21:12:24.564 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T21:12:24.564 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.565 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-202.gc7586e95.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-55.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T21:12:24.566 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-12-16T21:12:24.567 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:24.567 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-12-16T21:12:25.154 DEBUG:teuthology.parallel:result is None 2023-12-16T21:12:25.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:12:25.374 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T21:12:25.400 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-202.gc7586e95.el9 2023-12-16T21:12:25.401 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-202.gc7586e95.el9 2023-12-16T21:12:25.401 INFO:teuthology.task.install:The correct ceph version 19.0.0-202.gc7586e95 is installed. 2023-12-16T21:12:25.403 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:12:25.622 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T21:12:25.647 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-202.gc7586e95.el9 2023-12-16T21:12:25.647 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-202.gc7586e95.el9 2023-12-16T21:12:25.647 INFO:teuthology.task.install:The correct ceph version 19.0.0-202.gc7586e95 is installed. 2023-12-16T21:12:25.648 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-16T21:12:25.649 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:25.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T21:12:25.687 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:25.688 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T21:12:25.720 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-16T21:12:25.720 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:25.720 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T21:12:25.763 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T21:12:25.836 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:25.837 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T21:12:25.866 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T21:12:25.935 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-16T21:12:25.935 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:25.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T21:12:25.969 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T21:12:26.044 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:26.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T21:12:26.073 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T21:12:26.141 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-16T21:12:26.141 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:26.141 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T21:12:26.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T21:12:26.249 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:26.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T21:12:26.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T21:12:26.345 INFO:teuthology.run_tasks:Running task ceph... 2023-12-16T21:12:26.408 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-16T21:12:26.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2023-12-16T21:12:26.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2023-12-16T21:12:26.443 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-16T21:12:26.443 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-16T21:12:26.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-16T21:12:26.517 INFO:tasks.ceph:Creating extra log directories... 2023-12-16T21:12:26.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-16T21:12:26.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-16T21:12:26.596 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-16T21:12:26.596 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-16T21:12:26.596 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-16T21:12:26.596 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:26.596 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-16T21:12:26.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-16T21:12:26.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-12-16T21:12:26.810 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:26.810 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-16T21:12:26.828 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-16T21:12:26.944 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-12-16T21:12:26.977 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-16T21:12:26.977 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-16T21:12:26.977 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833', 'branch': 'wip-yuri8-testing-2023-12-15-0911', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7494833', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri8-testing-2023-12-15-0911'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri8-testing-2023-12-15-0911', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2960, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri8-testing-2023-12-15-0911', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1a5SboxJoSrpG9mBywr6CCPu5P8Z3x4NaoXLuZ0bf4KeJcCsXygvslEt6eyjzJZfTB9xRERW+QgLRGV0099aw=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbkN8s8lHKeTZqUwEuj1Kn16eqSfxnbUrx/iZfNDoz3x1UwK7+tQOL2YiAs2mO3QT6yiHfQCZ6uKmanhTc6OTk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-16_15:45:14', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-16T21:12:26.978 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-16T21:12:26.980 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-16T21:12:27.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T21:12:27.035 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2023-12-16T21:12:27.094 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:27.094 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T21:12:27.120 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T21:12:27.120 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2023-12-16T21:12:27.176 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T21:12:27.176 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:27.176 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 803 Links: 1 2023-12-16T21:12:27.177 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:27.177 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:27.177 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-12-16 21:12:10.322362559 +0000 2023-12-16T21:12:27.177 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-12-16 21:10:15.606991858 +0000 2023-12-16T21:12:27.177 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-12-16 21:10:15.606991858 +0000 2023-12-16T21:12:27.177 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-12-16 21:10:15.606991858 +0000 2023-12-16T21:12:27.178 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T21:12:27.245 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-12-16T21:12:27.246 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-12-16T21:12:27.246 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 8.4856e-05 s, 6.0 MB/s 2023-12-16T21:12:27.247 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T21:12:27.303 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2023-12-16T21:12:27.358 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 822 Links: 1 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-12-16 21:12:10.323362549 +0000 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-12-16 21:10:16.109987242 +0000 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-12-16 21:10:16.109987242 +0000 2023-12-16T21:12:27.359 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-12-16 21:10:16.109987242 +0000 2023-12-16T21:12:27.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T21:12:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-12-16T21:12:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-12-16T21:12:27.428 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 8.1406e-05 s, 6.3 MB/s 2023-12-16T21:12:27.429 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T21:12:27.485 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2023-12-16T21:12:27.540 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 841 Links: 1 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-12-16 21:12:10.323362549 +0000 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-12-16 21:10:16.607982671 +0000 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-12-16 21:10:16.607982671 +0000 2023-12-16T21:12:27.541 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-12-16 21:10:16.607982671 +0000 2023-12-16T21:12:27.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T21:12:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-12-16T21:12:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-12-16T21:12:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 7.6754e-05 s, 6.7 MB/s 2023-12-16T21:12:27.610 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T21:12:27.666 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2023-12-16T21:12:27.721 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T21:12:27.721 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 862 Links: 1 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2023-12-16 21:12:10.323362549 +0000 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2023-12-16 21:10:17.116978000 +0000 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2023-12-16 21:10:17.116978000 +0000 2023-12-16T21:12:27.722 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2023-12-16 21:10:17.116978000 +0000 2023-12-16T21:12:27.722 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T21:12:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2023-12-16T21:12:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2023-12-16T21:12:27.790 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 7.6543e-05 s, 6.7 MB/s 2023-12-16T21:12:27.792 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T21:12:27.848 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:27.848 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:27.848 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T21:12:27.866 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T21:12:27.867 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2023-12-16T21:12:27.925 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T21:12:27.925 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:27.925 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-16T21:12:27.925 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:27.925 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:27.925 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-12-16 21:12:17.734657355 +0000 2023-12-16T21:12:27.926 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-12-16 21:09:55.844797630 +0000 2023-12-16T21:12:27.926 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-12-16 21:09:55.844797630 +0000 2023-12-16T21:12:27.926 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2023-12-16 21:09:55.844797630 +0000 2023-12-16T21:12:27.926 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T21:12:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-12-16T21:12:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-12-16T21:12:27.994 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000195671 s, 2.6 MB/s 2023-12-16T21:12:27.995 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T21:12:28.052 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2023-12-16T21:12:28.109 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-12-16 21:12:17.734657355 +0000 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-12-16 21:09:56.343791243 +0000 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-12-16 21:09:56.343791243 +0000 2023-12-16T21:12:28.110 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2023-12-16 21:09:56.343791243 +0000 2023-12-16T21:12:28.111 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T21:12:28.177 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-12-16T21:12:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-12-16T21:12:28.178 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000217401 s, 2.4 MB/s 2023-12-16T21:12:28.179 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T21:12:28.235 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-12-16 21:12:17.734657355 +0000 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-12-16 21:09:56.828785038 +0000 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-12-16 21:09:56.828785038 +0000 2023-12-16T21:12:28.292 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2023-12-16 21:09:56.828785038 +0000 2023-12-16T21:12:28.293 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T21:12:28.359 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-12-16T21:12:28.359 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-12-16T21:12:28.359 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000215136 s, 2.4 MB/s 2023-12-16T21:12:28.360 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T21:12:28.416 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2023-12-16T21:12:28.471 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T21:12:28.471 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:12:28.471 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-16T21:12:28.472 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:12:28.472 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:12:28.472 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-12-16 21:12:17.735657342 +0000 2023-12-16T21:12:28.472 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-12-16 21:09:57.313778835 +0000 2023-12-16T21:12:28.472 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-12-16 21:09:57.313778835 +0000 2023-12-16T21:12:28.472 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2023-12-16 21:09:57.313778835 +0000 2023-12-16T21:12:28.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T21:12:28.537 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-12-16T21:12:28.537 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-12-16T21:12:28.537 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000182968 s, 2.8 MB/s 2023-12-16T21:12:28.538 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T21:12:28.595 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:28.595 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-16T21:12:28.596 INFO:tasks.ceph:Generating config... 2023-12-16T21:12:28.597 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-16T21:12:28.597 INFO:tasks.ceph:[client] debug client = 20 2023-12-16T21:12:28.597 INFO:tasks.ceph:[client] debug ms = 1 2023-12-16T21:12:28.597 INFO:tasks.ceph:[client] pid file = 2023-12-16T21:12:28.597 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-16T21:12:28.597 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-16T21:12:28.597 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-16T21:12:28.597 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-12-16T21:12:28.597 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-16T21:12:28.597 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-16T21:12:28.597 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-16T21:12:28.598 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-16T21:12:28.598 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-16T21:12:28.598 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-16T21:12:28.598 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-16T21:12:28.598 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-16T21:12:28.599 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-16T21:12:28.599 INFO:tasks.ceph:Setting up mon.a... 2023-12-16T21:12:28.600 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-16T21:12:28.658 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2023-12-16T21:12:28.660 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-16T21:12:28.709 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-16T21:12:28.740 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46'), ('mon.b', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]')] 2023-12-16T21:12:28.740 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-16T21:12:28.741 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:28.741 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T21:12:28.796 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --add b 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:28.881 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid 2397e6ee-7ffe-4438-b275-b4e10dea2d2f 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = pacific 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:fsid 2397e6ee-7ffe-4438-b275-b4e10dea2d2f 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2023-12-16T21:12:28.882914+0000 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:created 2023-12-16T21:12:28.882914+0000 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 16 (pacific) 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:1: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2023-12-16T21:12:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2023-12-16T21:12:28.883 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-16T21:12:28.884 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-16T21:12:28.898 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2397e6ee-7ffe-4438-b275-b4e10dea2d2f... 2023-12-16T21:12:28.899 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-16T21:12:28.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-16T21:12:28.993 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2023-12-16T21:12:28.993 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2023-12-16T21:12:28.993 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T21:12:28.993 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2023-12-16T21:12:28.993 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T21:12:28.994 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T21:12:28.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.46,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 2397e6ee-7ffe-4438-b275-b4e10dea2d2f 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2023-12-16T21:12:28.996 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.997 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2023-12-16T21:12:28.998 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:12:28.999 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T21:12:29.000 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2023-12-16T21:12:29.001 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2023-12-16T21:12:29.002 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2023-12-16T21:12:29.003 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = cephx 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2023-12-16T21:12:29.004 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2023-12-16T21:12:29.005 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2023-12-16T21:12:29.006 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = "172.21.15.46,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = 2397e6ee-7ffe-4438-b275-b4e10dea2d2f 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2023-12-16T21:12:29.007 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2023-12-16T21:12:29.008 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: bdev async discard = True 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: bdev enable discard = True 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore allocator = bitmap 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 1/20 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 1/20 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 20 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 4/10 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2023-12-16T21:12:29.009 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 20 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2023-12-16T21:12:29.010 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2023-12-16T21:12:29.011 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: mon op complaint time = 120 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: client mount timeout = 600 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 20 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = "" 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2023-12-16T21:12:29.012 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.b] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.c] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout:[mds] 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 20 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 20 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug frag = True 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: mds debug scatterstat = True 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: mds op complaint time = 180 2023-12-16T21:12:29.013 INFO:teuthology.orchestra.run.smithi046.stdout: mds verify scatter = True 2023-12-16T21:12:29.014 INFO:teuthology.orchestra.run.smithi046.stdout: osd op complaint time = 180 2023-12-16T21:12:29.014 INFO:teuthology.orchestra.run.smithi046.stdout: rados mon op timeout = 900 2023-12-16T21:12:29.014 INFO:teuthology.orchestra.run.smithi046.stdout: rados osd op timeout = 900 2023-12-16T21:12:29.016 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-16T21:12:29.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-16T21:12:29.116 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-16T21:12:29.116 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:29.116 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-16T21:12:29.131 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:29.131 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-16T21:12:29.186 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:12:29.187 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:29.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-16T21:12:29.187 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-16T21:12:29.274 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:29.274 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:29.329 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:12:29.330 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:29.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-16T21:12:29.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-16T21:12:29.373 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:29.373 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:29.429 INFO:tasks.ceph:Setting up mon nodes... 2023-12-16T21:12:29.429 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-16T21:12:29.429 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-16T21:12:29.491 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-16T21:12:29.494 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-16T21:12:29.560 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-16T21:12:29.563 INFO:tasks.ceph:Setting up mds nodes... 2023-12-16T21:12:29.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-16T21:12:29.626 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-16T21:12:29.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-16T21:12:29.659 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-16T21:12:29.760 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-16T21:12:29.763 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-16T21:12:29.793 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-16T21:12:29.851 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-16T21:12:29.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-16T21:12:29.881 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-16T21:12:29.975 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-16T21:12:29.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-16T21:12:30.004 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-16T21:12:30.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-16T21:12:30.052 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-16T21:12:30.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-12-16T21:12:30.116 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-12-16T21:12:30.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-16T21:12:30.178 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-16T21:12:30.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-12-16T21:12:30.275 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-12-16T21:12:30.290 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-16T21:12:30.290 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-16T21:12:30.290 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-16T21:12:30.324 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:30.324 INFO:tasks.ceph:role: osd.0 2023-12-16T21:12:30.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:12:30.324 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-16T21:12:30.395 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:30.395 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:30.396 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:30.428 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-12-16T21:12:30.430 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-12-16T21:12:30.430 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-16T21:12:30.506 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-16T21:12:30.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-16T21:12:30.653 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:30.653 INFO:tasks.ceph:role: osd.1 2023-12-16T21:12:30.653 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:12:30.654 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-16T21:12:30.724 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:30.724 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:30.724 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:30.724 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:30.724 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:30.725 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:30.725 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:30.725 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:30.725 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:30.725 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:30.756 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-12-16T21:12:30.758 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-12-16T21:12:30.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-16T21:12:30.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-16T21:12:30.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-16T21:12:30.981 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:30.982 INFO:tasks.ceph:role: osd.2 2023-12-16T21:12:30.982 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:12:30.982 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-16T21:12:31.052 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:31.052 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:31.053 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:31.085 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-12-16T21:12:31.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-12-16T21:12:31.087 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-16T21:12:31.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-16T21:12:31.241 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-16T21:12:31.312 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:31.312 INFO:tasks.ceph:role: osd.3 2023-12-16T21:12:31.312 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:12:31.313 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-16T21:12:31.382 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:31.382 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:31.382 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:31.382 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:31.383 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:31.383 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:31.383 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:31.383 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:31.383 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:31.383 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:31.415 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2023-12-16T21:12:31.417 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2023-12-16T21:12:31.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-16T21:12:31.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-16T21:12:31.570 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:31.670+0000 7f88d8120500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-16T21:12:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:31.670+0000 7f88d8120500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-16T21:12:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:31.670+0000 7f88d8120500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-16T21:12:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:31.670+0000 7f88d8120500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-16T21:12:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:31.670+0000 7f88d8120500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-16T21:12:31.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:31.670+0000 7f88d8120500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-16T21:12:34.531 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-16T21:12:34.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:34.664+0000 7fc43fd39500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-16T21:12:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:34.664+0000 7fc43fd39500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-16T21:12:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:34.664+0000 7fc43fd39500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-16T21:12:34.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:34.664+0000 7fc43fd39500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-16T21:12:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:34.664+0000 7fc43fd39500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-16T21:12:34.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:34.665+0000 7fc43fd39500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-16T21:12:37.516 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-16T21:12:37.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:37.647+0000 7f5b1ce11500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-16T21:12:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:37.647+0000 7f5b1ce11500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-16T21:12:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:37.647+0000 7f5b1ce11500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-16T21:12:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:37.647+0000 7f5b1ce11500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-16T21:12:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:37.647+0000 7f5b1ce11500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-16T21:12:37.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:37.647+0000 7f5b1ce11500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-16T21:12:40.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-16T21:12:40.539 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:40.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:40.636+0000 7f610ec16500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-16T21:12:40.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:40.636+0000 7f610ec16500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-16T21:12:40.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:40.636+0000 7f610ec16500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-16T21:12:40.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:40.636+0000 7f610ec16500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-16T21:12:40.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:40.636+0000 7f610ec16500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-16T21:12:40.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:12:40.636+0000 7f610ec16500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-16T21:12:43.494 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-16T21:12:43.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-16T21:12:43.562 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:43.562 INFO:tasks.ceph:role: osd.4 2023-12-16T21:12:43.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:12:43.562 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:43.640 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:43.641 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:43.641 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:43.641 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:44.143 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-12-16T21:12:44.146 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-12-16T21:12:44.146 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-16T21:12:44.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-16T21:12:44.292 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-16T21:12:44.360 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:44.360 INFO:tasks.ceph:role: osd.5 2023-12-16T21:12:44.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:12:44.361 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-16T21:12:44.435 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:44.435 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:44.436 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:44.901 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-12-16T21:12:44.903 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-12-16T21:12:44.903 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-16T21:12:44.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-16T21:12:45.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-16T21:12:45.093 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:45.093 INFO:tasks.ceph:role: osd.6 2023-12-16T21:12:45.093 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:12:45.093 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:45.167 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:45.649 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-12-16T21:12:45.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-12-16T21:12:45.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-16T21:12:45.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-16T21:12:45.814 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-16T21:12:45.881 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-16T21:12:45.881 INFO:tasks.ceph:role: osd.7 2023-12-16T21:12:45.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:12:45.881 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-16T21:12:45.955 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-16T21:12:45.956 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-16T21:12:46.483 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-12-16T21:12:46.485 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-12-16T21:12:46.486 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-16T21:12:46.616 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-16T21:12:46.652 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:46.759+0000 7f8f4ea00500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-16T21:12:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:46.760+0000 7f8f4ea00500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-16T21:12:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:46.760+0000 7f8f4ea00500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-16T21:12:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:46.760+0000 7f8f4ea00500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-16T21:12:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:46.760+0000 7f8f4ea00500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-16T21:12:46.760 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:46.760+0000 7f8f4ea00500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-16T21:12:49.622 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-16T21:12:49.653 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:49.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:49.744+0000 7fa83a8a5500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-16T21:12:49.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:49.744+0000 7fa83a8a5500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-16T21:12:49.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:49.744+0000 7fa83a8a5500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-16T21:12:49.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:49.744+0000 7fa83a8a5500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-16T21:12:49.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:49.744+0000 7fa83a8a5500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-16T21:12:49.744 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:49.745+0000 7fa83a8a5500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-16T21:12:52.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-16T21:12:52.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:52.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:52.730+0000 7f89eeada500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-16T21:12:52.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:52.730+0000 7f89eeada500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-16T21:12:52.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:52.730+0000 7f89eeada500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-16T21:12:52.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:52.730+0000 7f89eeada500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-16T21:12:52.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:52.730+0000 7f89eeada500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-16T21:12:52.730 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:52.731+0000 7f89eeada500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-16T21:12:55.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-16T21:12:55.615 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:12:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:55.709+0000 7f23f3016500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-16T21:12:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:55.709+0000 7f23f3016500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-16T21:12:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:55.709+0000 7f23f3016500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-16T21:12:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:55.709+0000 7f23f3016500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-16T21:12:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:55.709+0000 7f23f3016500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-16T21:12:55.709 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:12:55.710+0000 7f23f3016500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-16T21:12:56.979 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:12:56.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:12:58.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-16T21:12:58.620 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-16T21:12:58.620 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:58.620 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-16T21:12:58.656 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:58.656 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-16T21:12:58.730 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:58.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-16T21:12:58.803 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:58.803 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-16T21:12:58.875 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:58.876 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-16T21:12:58.947 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:58.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-16T21:12:59.019 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:59.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-16T21:12:59.090 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-16T21:12:59.122 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.122 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-16T21:12:59.193 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.194 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-16T21:12:59.263 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-16T21:12:59.331 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-16T21:12:59.399 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.399 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-16T21:12:59.466 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.467 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-16T21:12:59.534 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:59.534 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-16T21:12:59.551 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:12:59.551 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-12-16T21:12:59.608 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.608 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-16T21:12:59.623 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:12:59.623 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-12-16T21:12:59.679 INFO:tasks.ceph:Adding keys to all mons... 2023-12-16T21:12:59.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-16T21:12:59.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-16T21:12:59.723 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.y] 2023-12-16T21:12:59.723 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC9En5lgvZiHRAA87oDIk7xwjmYe4gMMg1NTQ== 2023-12-16T21:12:59.723 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a] 2023-12-16T21:12:59.723 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC9En5lilFjJRAAS3jGgFJPZWkcXnfWFuSkmw== 2023-12-16T21:12:59.724 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c] 2023-12-16T21:12:59.724 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC9En5lElZsLRAA5EGpXLzgGQdhS3J+Pf+9gw== 2023-12-16T21:12:59.724 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2023-12-16T21:12:59.724 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC/En5lteX6JxAAOUBR8wDo1UbdDenhHLeSNg== 2023-12-16T21:12:59.724 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2023-12-16T21:12:59.724 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDCEn5le0WlJxAAeyeKmj4yjwDCNn0WuoW4Rw== 2023-12-16T21:12:59.725 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2023-12-16T21:12:59.725 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDFEn5l/ZafJhAA72K0MtXsz0kQAa0Hhlrg5g== 2023-12-16T21:12:59.725 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.3] 2023-12-16T21:12:59.725 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDIEn5l/BX4JRAAcfb0TNdwljS3Hwp6E3EPXw== 2023-12-16T21:12:59.725 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2023-12-16T21:12:59.725 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC9En5lD2F+IRAAD5Xq3cx5Gd9fKtnuYe9HMw== 2023-12-16T21:12:59.726 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b] 2023-12-16T21:12:59.726 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC9En5liiXVMhAAJWJpsjXpIJHc/8ySyArSgQ== 2023-12-16T21:12:59.726 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d] 2023-12-16T21:12:59.726 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC9En5l7uY3OhAAwqyF6WkAH8BGy1tWJygLCg== 2023-12-16T21:12:59.726 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.4] 2023-12-16T21:12:59.727 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDOEn5lmZNQLRAAZ4THlfKtWJxeQXvoVShsRg== 2023-12-16T21:12:59.727 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.5] 2023-12-16T21:12:59.727 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDREn5lPZlmLBAABoR8+M5fdPUah5hZITIABA== 2023-12-16T21:12:59.727 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.6] 2023-12-16T21:12:59.727 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDUEn5lkOKRKxAAa5eJiAzXy/L79xL7egwC+g== 2023-12-16T21:12:59.728 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.7] 2023-12-16T21:12:59.728 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDXEn5lyQ1RKhAAcFWeiQsS4BaAJ/fIgCiCPA== 2023-12-16T21:12:59.728 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:12:59.728 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:12:59.728 INFO:teuthology.orchestra.run.smithi046.stdout:[client.2] 2023-12-16T21:12:59.728 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5l66kFBxAApn9h5R4KlGm9Udlz7FGldw== 2023-12-16T21:12:59.729 INFO:teuthology.orchestra.run.smithi046.stdout:[client.1] 2023-12-16T21:12:59.729 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5l7si2ChAAPixHXNSDvbDoEOOjVYonbQ== 2023-12-16T21:12:59.729 INFO:teuthology.orchestra.run.smithi046.stdout:[client.3] 2023-12-16T21:12:59.729 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5l1NSAEBAAqlxrkimPL3Z1z3llw4xr9Q== 2023-12-16T21:12:59.744 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2023-12-16T21:12:59.744 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC9En5lgvZiHRAA87oDIk7xwjmYe4gMMg1NTQ== 2023-12-16T21:12:59.744 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC9En5lilFjJRAAS3jGgFJPZWkcXnfWFuSkmw== 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC9En5lElZsLRAA5EGpXLzgGQdhS3J+Pf+9gw== 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC/En5lteX6JxAAOUBR8wDo1UbdDenhHLeSNg== 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2023-12-16T21:12:59.745 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDCEn5le0WlJxAAeyeKmj4yjwDCNn0WuoW4Rw== 2023-12-16T21:12:59.746 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2023-12-16T21:12:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDFEn5l/ZafJhAA72K0MtXsz0kQAa0Hhlrg5g== 2023-12-16T21:12:59.746 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2023-12-16T21:12:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDIEn5l/BX4JRAAcfb0TNdwljS3Hwp6E3EPXw== 2023-12-16T21:12:59.746 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2023-12-16T21:12:59.746 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC9En5lD2F+IRAAD5Xq3cx5Gd9fKtnuYe9HMw== 2023-12-16T21:12:59.747 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-12-16T21:12:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC9En5liiXVMhAAJWJpsjXpIJHc/8ySyArSgQ== 2023-12-16T21:12:59.747 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2023-12-16T21:12:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC9En5l7uY3OhAAwqyF6WkAH8BGy1tWJygLCg== 2023-12-16T21:12:59.747 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2023-12-16T21:12:59.747 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDOEn5lmZNQLRAAZ4THlfKtWJxeQXvoVShsRg== 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDREn5lPZlmLBAABoR8+M5fdPUah5hZITIABA== 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDUEn5lkOKRKxAAa5eJiAzXy/L79xL7egwC+g== 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDXEn5lyQ1RKhAAcFWeiQsS4BaAJ/fIgCiCPA== 2023-12-16T21:12:59.748 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2023-12-16T21:12:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:12:59.749 INFO:teuthology.orchestra.run.smithi121.stdout:[client.2] 2023-12-16T21:12:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC+En5l66kFBxAApn9h5R4KlGm9Udlz7FGldw== 2023-12-16T21:12:59.749 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2023-12-16T21:12:59.749 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC+En5l7si2ChAAPixHXNSDvbDoEOOjVYonbQ== 2023-12-16T21:12:59.749 INFO:teuthology.orchestra.run.smithi121.stdout:[client.3] 2023-12-16T21:12:59.750 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC+En5l1NSAEBAAqlxrkimPL3Z1z3llw4xr9Q== 2023-12-16T21:12:59.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:12:59.773 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:12:59.843 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:12:59.846 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:12:59.904 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:12:59.907 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:12:59.964 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:12:59.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.022 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.080 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.134 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:13:00.193 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-16T21:13:00.247 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:13:00.250 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:13:00.307 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:13:00.309 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-16T21:13:00.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.422 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.480 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.536 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-16T21:13:00.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.591 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.709 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.769 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-16T21:13:00.826 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-16T21:13:00.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-16T21:13:00.856 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-16T21:13:00.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-16T21:13:00.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-16T21:13:01.014 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-16T21:13:01.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-16T21:13:01.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-16T21:13:01.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-16T21:13:01.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-16T21:13:01.328 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:13:01.330 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-16T21:13:01.382 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-16T21:13:01.382 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-16T21:13:01.382 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-16T21:13:01.386 INFO:tasks.ceph.mon.a:Started 2023-12-16T21:13:01.386 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-16T21:13:01.387 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-16T21:13:01.424 INFO:tasks.ceph.mon.b:Started 2023-12-16T21:13:01.424 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-16T21:13:01.424 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-16T21:13:01.426 INFO:tasks.ceph.mon.c:Started 2023-12-16T21:13:01.426 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-16T21:13:01.426 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-16T21:13:01.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-16T21:13:01.429 INFO:tasks.ceph.mgr.y:Started 2023-12-16T21:13:01.429 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-16T21:13:01.429 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-16T21:13:01.431 INFO:tasks.ceph.mgr.x:Started 2023-12-16T21:13:01.431 INFO:tasks.ceph:Setting crush tunables to default 2023-12-16T21:13:01.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2023-12-16T21:13:01.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.538+0000 7fad01bc2640 1 Processor -- start 2023-12-16T21:13:01.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.538+0000 7fad01bc2640 1 -- start start 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7fad01bc2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7fad01bc2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7fad01bc2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7fad01bc2640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15b2e0 con 0x7facfc15af00 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7fad01bc2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc0cf460 con 0x7facfc15d890 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7fad01bc2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc06d600 con 0x7facfc06cc50 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48468/0 (socket says 172.21.15.46:48468) 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 learned_addr learned my addr 172.21.15.46:0/3237351159 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:01.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:01.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.539+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:13:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:01.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:01.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:01.740+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:13:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:02.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:02.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.141+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:13:02.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:13:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:02.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:02.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:02.943+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:13:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 msgr2=0x7facfc06d0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7facfc06cc50 0x7facfc06d0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 msgr2=0x7facfc0cec10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7facfc15d890 0x7facfc0cec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:04.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 msgr2=0x7facfc0d08f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7facfc15af00 0x7facfc0d08f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facfc070180 con 0x7face4007340 2023-12-16T21:13:04.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:04.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face400c090 con 0x7face4003690 2023-12-16T21:13:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.539+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> v1:172.21.15.121:6789/0 conn(0x7face4003690 legacy=0x7face4005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.539+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> v1:172.21.15.46:6789/0 conn(0x7face4007340 legacy=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:07.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 reap_dead start 2023-12-16T21:13:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face400d270 con 0x7face4009f10 2023-12-16T21:13:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15b2e0 con 0x7face400c3b0 2023-12-16T21:13:07.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0cf460 con 0x7face400cdd0 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:07.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.540+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:07.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:07.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:07.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:13:07.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:07.741+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:08.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:08.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:08.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.142+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:13:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:08.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:08.943+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:13:10.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 msgr2=0x7face40097e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:10.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face400cdd0 0x7face40097e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:10.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 msgr2=0x7face400c7f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:10.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.121:3300/0 conn(0x7face400c3b0 0x7face400c7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 >> v1:172.21.15.46:6789/0 conn(0x7face4009f10 legacy=0x7face4005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face400a6b0 con 0x7face400aa90 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facf97fa640 1 -- 172.21.15.46:0/3237351159 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15b2e0 con 0x7face400b870 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:10.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.541+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:13:10.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.742+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.742+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.742+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.742+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:10.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:10.742+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:13:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.143+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.143+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-16T21:13:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.143+0000 7facfaffd640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-16T21:13:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.143+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-16T21:13:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.143+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:13:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.755+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770645783 0 0) 0x7face400a6b0 con 0x7face400aa90 2023-12-16T21:13:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.755+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face8003520 con 0x7face400aa90 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145891974 0 0) 0x7face8003520 con 0x7face400aa90 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 msgr2=0x7face40145a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 --2- 172.21.15.46:0/3237351159 >> v2:172.21.15.46:3300/0 conn(0x7face400b870 0x7face40145a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face400bde0 con 0x7face400aa90 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7facec0167e0 con 0x7face400aa90 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 >> v1:172.21.15.121:6789/0 conn(0x7face400aa90 legacy=0x7face400b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfbfff640 1 -- 172.21.15.46:0/3237351159 reap_dead start 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face8003970 0x7face8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7face8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face80052a0 0x7face800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc15b2e0 con 0x7face8003970 2023-12-16T21:13:11.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face800c150 con 0x7face80047c0 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face800c310 con 0x7face80052a0 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face8003970 0x7face8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.761+0000 7facfaffd640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7face8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.762+0000 7facfbfff640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face80052a0 0x7face800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.762+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face80052a0 msgr2=0x7face800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.762+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face80052a0 0x7face800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.762+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 msgr2=0x7face8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.762+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7face8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.762+0000 7facfb7fe640 1 -- 172.21.15.46:0/3237351159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8003520 con 0x7face8003970 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.763+0000 7facfb7fe640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face8003970 0x7face8003d70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7face4007250 tx=0x7face80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfceb0e9764ffc0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.763+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec02e780 con 0x7face8003970 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.763+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec00c040 con 0x7face8003970 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.763+0000 7facfa7fc640 1 -- 172.21.15.46:0/3237351159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec02eb00 con 0x7face8003970 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 -- 172.21.15.46:0/3237351159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face8003970 msgr2=0x7face8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face8003970 0x7face8003d70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7face4007250 tx=0x7face80057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 -- 172.21.15.46:0/3237351159 shutdown_connections 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face80052a0 0x7face800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7face8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 --2- 172.21.15.46:0/3237351159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face8003970 0x7face8003d70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 -- 172.21.15.46:0/3237351159 >> 172.21.15.46:0/3237351159 conn(0x7facfc0b2c10 msgr2=0x7facfc0b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 -- 172.21.15.46:0/3237351159 shutdown_connections 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.764+0000 7fad01bc2640 1 -- 172.21.15.46:0/3237351159 wait complete. 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.766+0000 7fad01bc2640 1 Processor -- start 2023-12-16T21:13:11.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.766+0000 7fad01bc2640 1 -- start start 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.767+0000 7fad01bc2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face8003970 0x7facfc0c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.767+0000 7fad01bc2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7facfc148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7fad01bc2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 0x7facfc06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7fad01bc2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0b5950 con 0x7face80052a0 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7fad01bc2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc0b5c30 con 0x7face80047c0 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7fad01bc2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc15ea60 con 0x7face8003970 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfb7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face8003970 0x7facfc0c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 0x7facfc06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfb7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face8003970 0x7facfc0c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:60444/0 (socket says 172.21.15.46:60444) 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfb7fe640 1 -- 172.21.15.46:0/1402466981 learned_addr learned my addr 172.21.15.46:0/1402466981 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 0x7facfc06bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44108/0 (socket says 172.21.15.46:44108) 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfaffd640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7facfc148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:11.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 -- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face8003970 msgr2=0x7facfc0c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face8003970 0x7facfc0c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 -- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 msgr2=0x7facfc148320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7facfc148320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 -- 172.21.15.46:0/1402466981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc15ed40 con 0x7face80052a0 2023-12-16T21:13:11.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.768+0000 7facfbfff640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 0x7facfc06bda0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facfc0b2ba0 tx=0x7facf4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.769+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4014a00 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.769+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facfc15f020 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.769+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf400f040 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.769+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facfc06c2e0 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.769+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4014da0 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.770+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7facf4017740 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.770+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facdc005250 con 0x7face80052a0 2023-12-16T21:13:11.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.771+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7facf40207d0 con 0x7face80052a0 2023-12-16T21:13:11.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.773+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7facf401b030 con 0x7face80052a0 2023-12-16T21:13:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.843+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7facdc0057d0 con 0x7face80052a0 2023-12-16T21:13:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.844+0000 7face3fff640 1 -- 172.21.15.46:0/1402466981 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7facf402d090 con 0x7face80052a0 2023-12-16T21:13:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2023-12-16T21:13:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.845+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 msgr2=0x7facfc06bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.845+0000 7fad01bc2640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 0x7facfc06bda0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facfc0b2ba0 tx=0x7facf4004750 comp rx=0 tx=0).stop 2023-12-16T21:13:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 shutdown_connections 2023-12-16T21:13:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7face80052a0 0x7facfc06bda0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7face80047c0 0x7facfc148320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 --2- 172.21.15.46:0/1402466981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7face8003970 0x7facfc0c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 >> 172.21.15.46:0/1402466981 conn(0x7facfc0b2c10 msgr2=0x7facfc15d120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 shutdown_connections 2023-12-16T21:13:11.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:11.846+0000 7fad01bc2640 1 -- 172.21.15.46:0/1402466981 wait complete. 2023-12-16T21:13:11.858 INFO:tasks.ceph:check_enable_crimson: False 2023-12-16T21:13:11.858 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-16T21:13:11.858 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:13:11.858 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-16T21:13:11.890 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:13:11.890 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-16T21:13:11.961 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:13:11.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-16T21:13:12.033 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:13:12.033 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-16T21:13:12.104 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:13:12.104 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-16T21:13:12.132 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:13:12.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-16T21:13:12.200 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:13:12.200 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-16T21:13:12.215 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:12.214+0000 7ff016433040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T21:13:12.240 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:13:12.241 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-16T21:13:12.252 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:12.251+0000 7f16ee04c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T21:13:12.268 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:12.268+0000 7ff016433040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T21:13:12.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 8849da25-21f9-457e-ad40-67ca2672b6bf 0 2023-12-16T21:13:12.306 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:12.306+0000 7f16ee04c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T21:13:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.434+0000 7f4d93e0e640 1 Processor -- start 2023-12-16T21:13:12.435 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d93e0e640 1 -- start start 2023-12-16T21:13:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d93e0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c06eee0 0x7f4d8c15b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d93e0e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c13e470 con 0x7f4d8c15f830 2023-12-16T21:13:12.438 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d93e0e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8c15c2f0 con 0x7f4d8c13e090 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d93e0e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c15c5d0 con 0x7f4d8c06eee0 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d913a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c06eee0 0x7f4d8c15b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d91baa640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4d8c13e090 0x7f4d8c06e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:56750/0 (socket says 172.21.15.121:56750) 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.435+0000 7f4d91baa640 1 -- 172.21.15.121:0/1939538195 learned_addr learned my addr 172.21.15.121:0/1939538195 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.436+0000 7f4d90ba8640 1 -- 172.21.15.121:0/1939538195 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978179969 0 0) 0x7f4d8c15c2f0 con 0x7f4d8c13e090 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.436+0000 7f4d90ba8640 1 -- 172.21.15.121:0/1939538195 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d78003520 con 0x7f4d8c13e090 2023-12-16T21:13:12.439 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.436+0000 7f4d913a9640 1 -- 172.21.15.121:0/1939538195 >> v1:172.21.15.121:6789/0 conn(0x7f4d8c13e090 legacy=0x7f4d8c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.436+0000 7f4d913a9640 1 -- 172.21.15.121:0/1939538195 >> v1:172.21.15.46:6789/0 conn(0x7f4d8c15f830 legacy=0x7f4d8c15bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.436+0000 7f4d913a9640 1 -- 172.21.15.121:0/1939538195 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c15ed50 con 0x7f4d8c06eee0 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.436+0000 7f4d913a9640 1 --2- 172.21.15.121:0/1939538195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c06eee0 0x7f4d8c15b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d880097d0 tx=0x7f4d8802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45e37b189bc06cde server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d90ba8640 1 -- 172.21.15.121:0/1939538195 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88031800 con 0x7f4d8c06eee0 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d90ba8640 1 -- 172.21.15.121:0/1939538195 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d88031960 con 0x7f4d8c06eee0 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d90ba8640 1 -- 172.21.15.121:0/1939538195 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d88031cf0 con 0x7f4d8c06eee0 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 -- 172.21.15.121:0/1939538195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c06eee0 msgr2=0x7f4d8c15b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 --2- 172.21.15.121:0/1939538195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c06eee0 0x7f4d8c15b5c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4d880097d0 tx=0x7f4d8802b8a0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 -- 172.21.15.121:0/1939538195 shutdown_connections 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 --2- 172.21.15.121:0/1939538195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c06eee0 0x7f4d8c15b5c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 -- 172.21.15.121:0/1939538195 >> 172.21.15.121:0/1939538195 conn(0x7f4d8c06a8c0 msgr2=0x7f4d8c06acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 -- 172.21.15.121:0/1939538195 shutdown_connections 2023-12-16T21:13:12.440 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.437+0000 7f4d93e0e640 1 -- 172.21.15.121:0/1939538195 wait complete. 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.438+0000 7f4d93e0e640 1 Processor -- start 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 -- start start 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 0x7f4d8c1400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c13e090 0x7f4d8c155b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c15f830 0x7f4d8c13a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0d0280 con 0x7f4d8c13e090 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c06e8b0 con 0x7f4d8c06eee0 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d93e0e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c06eb90 con 0x7f4d8c15f830 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d91baa640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 0x7f4d8c1400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d91baa640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 0x7f4d8c1400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47558/0 (socket says 172.21.15.121:47558) 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d91baa640 1 -- 172.21.15.121:0/3120961462 learned_addr learned my addr 172.21.15.121:0/3120961462 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:12.443 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d923ab640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c15f830 0x7f4d8c13a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.439+0000 7f4d913a9640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c13e090 0x7f4d8c155b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d91baa640 1 -- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c15f830 msgr2=0x7f4d8c13a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d91baa640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c15f830 0x7f4d8c13a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d91baa640 1 -- 172.21.15.121:0/3120961462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c13e090 msgr2=0x7f4d8c155b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d91baa640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c13e090 0x7f4d8c155b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d91baa640 1 -- 172.21.15.121:0/3120961462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d88009400 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d923ab640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c15f830 0x7f4d8c13a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d91baa640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 0x7f4d8c1400f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c500 tx=0x7f4d8000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80005ce0 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d80005e40 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c0d0490 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80007330 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c156090 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4d8001c440 con 0x7f4d8c06eee0 2023-12-16T21:13:12.444 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.440+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4d80012940 con 0x7f4d8c06eee0 2023-12-16T21:13:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.441+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c067420 con 0x7f4d8c06eee0 2023-12-16T21:13:12.445 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.443+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f4d80017070 con 0x7f4d8c06eee0 2023-12-16T21:13:12.516 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.515+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8849da25-21f9-457e-ad40-67ca2672b6bf", "id": 0} v 0) v1 -- 0x7f4d8c1591e0 con 0x7f4d8c06eee0 2023-12-16T21:13:12.532 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.518+0000 7f4d7e7fc640 1 -- 172.21.15.121:0/3120961462 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8849da25-21f9-457e-ad40-67ca2672b6bf", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4d80012c20 con 0x7f4d8c06eee0 2023-12-16T21:13:12.532 INFO:teuthology.orchestra.run.smithi121.stdout:0 2023-12-16T21:13:12.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 msgr2=0x7f4d8c1400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 0x7f4d8c1400f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c500 tx=0x7f4d8000c9c0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 shutdown_connections 2023-12-16T21:13:12.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c13e090 0x7f4d8c155b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d8c15f830 0x7f4d8c13a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 --2- 172.21.15.121:0/3120961462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d8c06eee0 0x7f4d8c1400f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 >> 172.21.15.121:0/3120961462 conn(0x7f4d8c06a8c0 msgr2=0x7f4d8c0b2390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:12.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 shutdown_connections 2023-12-16T21:13:12.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.519+0000 7f4d93e0e640 1 -- 172.21.15.121:0/3120961462 wait complete. 2023-12-16T21:13:12.537 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 2cbc7e60-c77c-40e8-b366-7213dbacbd92 1 2023-12-16T21:13:12.636 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.636+0000 7efc066c9640 1 Processor -- start 2023-12-16T21:13:12.637 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.636+0000 7efc066c9640 1 -- start start 2023-12-16T21:13:12.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.637+0000 7efc066c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efc0006dc70 0x7efc0015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.637+0000 7efc066c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc0015e220 0x7efc000ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.638 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.637+0000 7efc066c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0013d440 con 0x7efc0013d060 2023-12-16T21:13:12.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.637+0000 7efc066c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0015eaa0 con 0x7efc0006dc70 2023-12-16T21:13:12.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.637+0000 7efc066c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0015a700 con 0x7efc0015e220 2023-12-16T21:13:12.639 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efc04c66640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc0015e220 0x7efc000ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efbfffff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efc0013d060 0x7efc0006d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:56766/0 (socket says 172.21.15.121:56766) 2023-12-16T21:13:12.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efc04c66640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc0015e220 0x7efc000ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47572/0 (socket says 172.21.15.121:47572) 2023-12-16T21:13:12.648 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efbfffff640 1 -- 172.21.15.121:0/2744316928 learned_addr learned my addr 172.21.15.121:0/2744316928 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-12-16T21:13:12.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efc04c66640 1 -- 172.21.15.121:0/2744316928 >> v2:172.21.15.46:3300/0 conn(0x7efc0006dc70 msgr2=0x7efc0015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:12.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efbff7fe640 1 --2- 172.21.15.121:0/2744316928 >> v2:172.21.15.46:3300/0 conn(0x7efc0006dc70 0x7efc0015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efc04c66640 1 --2- 172.21.15.121:0/2744316928 >> v2:172.21.15.46:3300/0 conn(0x7efc0006dc70 0x7efc0015dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.649 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efbfeffd640 1 -- 172.21.15.121:0/2744316928 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301414502 0 0) 0x7efc0013d440 con 0x7efc0013d060 2023-12-16T21:13:12.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efc04c66640 1 -- 172.21.15.121:0/2744316928 >> v1:172.21.15.121:6789/0 conn(0x7efc0013d060 legacy=0x7efc0006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.638+0000 7efc04c66640 1 -- 172.21.15.121:0/2744316928 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0015a9e0 con 0x7efc0015e220 2023-12-16T21:13:12.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc04c66640 1 --2- 172.21.15.121:0/2744316928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0015e220 0x7efc000ceaa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efc00072820 tx=0x7efbf802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eabf70a3d21e4c7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:12.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efbfeffd640 1 -- 172.21.15.121:0/2744316928 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf802cc80 con 0x7efc0015e220 2023-12-16T21:13:12.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efbfeffd640 1 -- 172.21.15.121:0/2744316928 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf802cde0 con 0x7efc0015e220 2023-12-16T21:13:12.650 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efbfeffd640 1 -- 172.21.15.121:0/2744316928 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8034660 con 0x7efc0015e220 2023-12-16T21:13:12.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc066c9640 1 -- 172.21.15.121:0/2744316928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0015e220 msgr2=0x7efc000ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc066c9640 1 --2- 172.21.15.121:0/2744316928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0015e220 0x7efc000ceaa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efc00072820 tx=0x7efbf802bed0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc066c9640 1 -- 172.21.15.121:0/2744316928 shutdown_connections 2023-12-16T21:13:12.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc066c9640 1 --2- 172.21.15.121:0/2744316928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0015e220 0x7efc000ceaa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc066c9640 1 --2- 172.21.15.121:0/2744316928 >> v2:172.21.15.46:3300/0 conn(0x7efc0006dc70 0x7efc0015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.651 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.639+0000 7efc066c9640 1 -- 172.21.15.121:0/2744316928 >> 172.21.15.121:0/2744316928 conn(0x7efc00066270 msgr2=0x7efc00066650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:12.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.640+0000 7efc066c9640 1 -- 172.21.15.121:0/2744316928 shutdown_connections 2023-12-16T21:13:12.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.640+0000 7efc066c9640 1 -- 172.21.15.121:0/2744316928 wait complete. 2023-12-16T21:13:12.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.641+0000 7efc066c9640 1 Processor -- start 2023-12-16T21:13:12.652 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.641+0000 7efc066c9640 1 -- start start 2023-12-16T21:13:12.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.641+0000 7efc066c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 0x7efc000b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc0013d060 0x7efc000b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc0015e220 0x7efc00139520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001f1180 con 0x7efc0013d060 2023-12-16T21:13:12.653 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc001f13f0 con 0x7efc0006dc70 2023-12-16T21:13:12.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc001f16d0 con 0x7efc0015e220 2023-12-16T21:13:12.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 0x7efc000b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc0015e220 0x7efc00139520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfffff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 0x7efc000b4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47586/0 (socket says 172.21.15.121:47586) 2023-12-16T21:13:12.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfffff640 1 -- 172.21.15.121:0/2738181807 learned_addr learned my addr 172.21.15.121:0/2738181807 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:12.654 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbff7fe640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc0013d060 0x7efc000b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 -- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 msgr2=0x7efc000b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 0x7efc000b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 -- 172.21.15.121:0/2738181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc0013d060 msgr2=0x7efc000b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc0013d060 0x7efc000b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.655 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 -- 172.21.15.121:0/2738181807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf8009a70 con 0x7efc0015e220 2023-12-16T21:13:12.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfffff640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 0x7efc000b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:12.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc04c66640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc0015e220 0x7efc00139520 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efbf8033000 tx=0x7efbf8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:12.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8040650 con 0x7efc0015e220 2023-12-16T21:13:12.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf80407b0 con 0x7efc0015e220 2023-12-16T21:13:12.656 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc001f19b0 con 0x7efc0015e220 2023-12-16T21:13:12.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8040b60 con 0x7efc0015e220 2023-12-16T21:13:12.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.642+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc00136420 con 0x7efc0015e220 2023-12-16T21:13:12.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.643+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efbf8049400 con 0x7efc0015e220 2023-12-16T21:13:12.657 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.643+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7efbf803e7b0 con 0x7efc0015e220 2023-12-16T21:13:12.658 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.643+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc000afc50 con 0x7efc0015e220 2023-12-16T21:13:12.658 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.645+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7efbf8067cf0 con 0x7efc0015e220 2023-12-16T21:13:12.717 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.716+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2cbc7e60-c77c-40e8-b366-7213dbacbd92", "id": 1} v 0) v1 -- 0x7efc00067420 con 0x7efc0015e220 2023-12-16T21:13:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.719+0000 7efbfcff9640 1 -- 172.21.15.121:0/2738181807 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2cbc7e60-c77c-40e8-b366-7213dbacbd92", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7efbf8056030 con 0x7efc0015e220 2023-12-16T21:13:12.720 INFO:teuthology.orchestra.run.smithi121.stdout:1 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc0015e220 msgr2=0x7efc00139520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc0015e220 0x7efc00139520 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7efbf8033000 tx=0x7efbf8004a40 comp rx=0 tx=0).stop 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 shutdown_connections 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc0013d060 0x7efc000b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc0015e220 0x7efc00139520 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 --2- 172.21.15.121:0/2738181807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0006dc70 0x7efc000b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 >> 172.21.15.121:0/2738181807 conn(0x7efc00066270 msgr2=0x7efc000b0820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 shutdown_connections 2023-12-16T21:13:12.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.720+0000 7efc066c9640 1 -- 172.21.15.121:0/2738181807 wait complete. 2023-12-16T21:13:12.734 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 017ec04d-f0fd-4898-b169-0571b4ca3fe5 2 2023-12-16T21:13:12.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.835+0000 7ff0c5bb0640 1 Processor -- start 2023-12-16T21:13:12.836 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.835+0000 7ff0c5bb0640 1 -- start start 2023-12-16T21:13:12.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0c5bb0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c013d060 0x7ff0c006d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0c5bb0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 0x7ff0c015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0c5bb0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff0c015e220 0x7ff0c0071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.850 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0c5bb0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c006df20 con 0x7ff0c015e220 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0c5bb0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c013d440 con 0x7ff0c006daa0 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0c5bb0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c015eaa0 con 0x7ff0c013d060 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0beffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 0x7ff0c015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0bf7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c013d060 0x7ff0c006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0beffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 0x7ff0c015dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47592/0 (socket says 172.21.15.121:47592) 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0beffd640 1 -- 172.21.15.121:0/440660403 learned_addr learned my addr 172.21.15.121:0/440660403 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.836+0000 7ff0bffff640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.46:3300/0 conn(0x7ff0c015e220 0x7ff0c0071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bf7fe640 1 -- 172.21.15.121:0/440660403 >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 msgr2=0x7ff0c015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:12.851 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bf7fe640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 0x7ff0c015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bf7fe640 1 -- 172.21.15.121:0/440660403 >> v2:172.21.15.46:3300/0 conn(0x7ff0c015e220 msgr2=0x7ff0c0071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bf7fe640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.46:3300/0 conn(0x7ff0c015e220 0x7ff0c0071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bf7fe640 1 -- 172.21.15.121:0/440660403 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c015a700 con 0x7ff0c013d060 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0beffd640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 0x7ff0c015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bffff640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.46:3300/0 conn(0x7ff0c015e220 0x7ff0c0071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0bf7fe640 1 --2- 172.21.15.121:0/440660403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c013d060 0x7ff0c006d560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0072670 tx=0x7ff0b002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de54808e02394797 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:12.852 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.837+0000 7ff0be7fc640 1 -- 172.21.15.121:0/440660403 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b002e070 con 0x7ff0c013d060 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0be7fc640 1 -- 172.21.15.121:0/440660403 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b00315a0 con 0x7ff0c013d060 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0be7fc640 1 -- 172.21.15.121:0/440660403 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0031930 con 0x7ff0c013d060 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/440660403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c013d060 msgr2=0x7ff0c006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/440660403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c013d060 0x7ff0c006d560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0072670 tx=0x7ff0b002b840 comp rx=0 tx=0).stop 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/440660403 shutdown_connections 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.46:3300/0 conn(0x7ff0c015e220 0x7ff0c0071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/440660403 >> v2:172.21.15.121:3300/0 conn(0x7ff0c006daa0 0x7ff0c015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.853 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/440660403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c013d060 0x7ff0c006d560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/440660403 >> 172.21.15.121:0/440660403 conn(0x7ff0c0066270 msgr2=0x7ff0c0066650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/440660403 shutdown_connections 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.838+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/440660403 wait complete. 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.839+0000 7ff0c5bb0640 1 Processor -- start 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 -- start start 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c006daa0 0x7ff0c0152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.854 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0c013d060 0x7ff0c0154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 0x7ff0c00b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c013bfe0 con 0x7ff0c006daa0 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01f6b70 con 0x7ff0c013d060 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0c5bb0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c01f6e50 con 0x7ff0c015e220 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0bffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 0x7ff0c00b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0beffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0c013d060 0x7ff0c0154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.855 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0bffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 0x7ff0c00b4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44504/0 (socket says 172.21.15.121:44504) 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0bffff640 1 -- 172.21.15.121:0/3173355259 learned_addr learned my addr 172.21.15.121:0/3173355259 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.840+0000 7ff0bf7fe640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c006daa0 0x7ff0c0152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0beffd640 1 -- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 msgr2=0x7ff0c00b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0beffd640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 0x7ff0c00b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0beffd640 1 -- 172.21.15.121:0/3173355259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c006daa0 msgr2=0x7ff0c0152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0beffd640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c006daa0 0x7ff0c0152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0beffd640 1 -- 172.21.15.121:0/3173355259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80094d0 con 0x7ff0c013d060 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0bffff640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 0x7ff0c00b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:12.856 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0beffd640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0c013d060 0x7ff0c0154780 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800e890 tx=0x7ff0a800ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800c940 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a800caa0 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800ce10 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0009400 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c01f7410 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff0a8013770 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.841+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7ff0a801cc80 con 0x7ff0c013d060 2023-12-16T21:13:12.857 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.842+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c0067420 con 0x7ff0c013d060 2023-12-16T21:13:12.858 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.844+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7ff0a8021060 con 0x7ff0c013d060 2023-12-16T21:13:12.863 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:12.862+0000 7ff016433040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T21:13:12.871 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:12.871+0000 7f16ee04c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T21:13:12.916 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:12.916+0000 7ff016433040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T21:13:12.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.916+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "017ec04d-f0fd-4898-b169-0571b4ca3fe5", "id": 2} v 0) v1 -- 0x7ff0c00b2c10 con 0x7ff0c013d060 2023-12-16T21:13:12.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.919+0000 7ff0b7fff640 1 -- 172.21.15.121:0/3173355259 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "017ec04d-f0fd-4898-b169-0571b4ca3fe5", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff0a8017070 con 0x7ff0c013d060 2023-12-16T21:13:12.920 INFO:teuthology.orchestra.run.smithi121.stdout:2 2023-12-16T21:13:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0c013d060 msgr2=0x7ff0c0154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0c013d060 0x7ff0c0154780 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800e890 tx=0x7ff0a800ed50 comp rx=0 tx=0).stop 2023-12-16T21:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 shutdown_connections 2023-12-16T21:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0c015e220 0x7ff0c00b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0c013d060 0x7ff0c0154780 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 --2- 172.21.15.121:0/3173355259 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0c006daa0 0x7ff0c0152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 >> 172.21.15.121:0/3173355259 conn(0x7ff0c0066270 msgr2=0x7ff0c0071550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 shutdown_connections 2023-12-16T21:13:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:12.920+0000 7ff0c5bb0640 1 -- 172.21.15.121:0/3173355259 wait complete. 2023-12-16T21:13:12.926 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:12.926+0000 7f16ee04c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T21:13:12.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 5194ffa9-d0bb-4150-bfef-dafd3a848ec8 3 2023-12-16T21:13:12.964 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:12.964+0000 7ff016433040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T21:13:12.973 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:12.972+0000 7f16ee04c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T21:13:13.019 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.019+0000 7ff016433040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T21:13:13.029 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.028+0000 7f16ee04c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T21:13:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.031+0000 7f03c5ddc640 1 Processor -- start 2023-12-16T21:13:13.032 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.031+0000 7f03c5ddc640 1 -- start start 2023-12-16T21:13:13.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03c5ddc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c006d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03c5ddc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 0x7f03c015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.035 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03c5ddc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f03c015e220 0x7f03c0071550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03c5ddc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c006df20 con 0x7f03c015e220 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03c5ddc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c013d440 con 0x7f03c006daa0 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03c5ddc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c015eaa0 con 0x7f03c013d060 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03b7fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 0x7f03c015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c006d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44518/0 (socket says 172.21.15.121:44518) 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 -- 172.21.15.121:0/1471510974 learned_addr learned my addr 172.21.15.121:0/1471510974 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03b7fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 0x7f03c015dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47606/0 (socket says 172.21.15.121:47606) 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bffff640 1 --2- 172.21.15.121:0/1471510974 >> v2:172.21.15.46:3300/0 conn(0x7f03c015e220 0x7f03c0071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.036 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 -- 172.21.15.121:0/1471510974 >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 msgr2=0x7f03c015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 --2- 172.21.15.121:0/1471510974 >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 0x7f03c015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 -- 172.21.15.121:0/1471510974 >> v2:172.21.15.46:3300/0 conn(0x7f03c015e220 msgr2=0x7f03c0071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 --2- 172.21.15.121:0/1471510974 >> v2:172.21.15.46:3300/0 conn(0x7f03c015e220 0x7f03c0071550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03bf7fe640 1 -- 172.21.15.121:0/1471510974 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c015a700 con 0x7f03c013d060 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.032+0000 7f03b7fff640 1 --2- 172.21.15.121:0/1471510974 >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 0x7f03c015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.033+0000 7f03bf7fe640 1 --2- 172.21.15.121:0/1471510974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c006d560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03c0072670 tx=0x7f03b002b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8edaaf0f32cda569 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.033+0000 7f03beffd640 1 -- 172.21.15.121:0/1471510974 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0031860 con 0x7f03c013d060 2023-12-16T21:13:13.037 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.033+0000 7f03beffd640 1 -- 172.21.15.121:0/1471510974 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b00319c0 con 0x7f03c013d060 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.033+0000 7f03beffd640 1 -- 172.21.15.121:0/1471510974 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0031d50 con 0x7f03c013d060 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 -- 172.21.15.121:0/1471510974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 msgr2=0x7f03c006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/1471510974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c006d560 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f03c0072670 tx=0x7f03b002b900 comp rx=0 tx=0).stop 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 -- 172.21.15.121:0/1471510974 shutdown_connections 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/1471510974 >> v2:172.21.15.46:3300/0 conn(0x7f03c015e220 0x7f03c0071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/1471510974 >> v2:172.21.15.121:3300/0 conn(0x7f03c006daa0 0x7f03c015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/1471510974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c006d560 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 -- 172.21.15.121:0/1471510974 >> 172.21.15.121:0/1471510974 conn(0x7f03c0066270 msgr2=0x7f03c0066650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.038 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 -- 172.21.15.121:0/1471510974 shutdown_connections 2023-12-16T21:13:13.039 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.034+0000 7f03c5ddc640 1 -- 172.21.15.121:0/1471510974 wait complete. 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.035+0000 7f03c5ddc640 1 Processor -- start 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.035+0000 7f03c5ddc640 1 -- start start 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03c5ddc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 0x7f03c0159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03c5ddc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03c5ddc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c015e220 0x7f03c00aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03c5ddc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c0132020 con 0x7f03c015e220 2023-12-16T21:13:13.042 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03c5ddc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c014d8a0 con 0x7f03c006daa0 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03c5ddc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c014db00 con 0x7f03c013d060 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 0x7f03c0159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 0x7f03c0159cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47612/0 (socket says 172.21.15.121:47612) 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 -- 172.21.15.121:0/2242665830 learned_addr learned my addr 172.21.15.121:0/2242665830 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03b7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bffff640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c015e220 0x7f03c00aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 -- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 msgr2=0x7f03c00b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c00b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.043 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 -- 172.21.15.121:0/2242665830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c015e220 msgr2=0x7f03c00aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c015e220 0x7f03c00aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.036+0000 7f03bf7fe640 1 -- 172.21.15.121:0/2242665830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a800b6a0 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03bf7fe640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 0x7f03c0159cf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03b002b940 tx=0x7f03b00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b003ac60 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b003ade0 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b00094d0 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b0044630 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c014e0c0 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f03b0044a90 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.037+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f03b0030840 con 0x7f03c006daa0 2023-12-16T21:13:13.044 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.038+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c0067420 con 0x7f03c006daa0 2023-12-16T21:13:13.045 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.040+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f03b0035020 con 0x7f03c006daa0 2023-12-16T21:13:13.064 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.063+0000 7ff016433040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T21:13:13.073 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.072+0000 7f16ee04c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T21:13:13.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.111+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5194ffa9-d0bb-4150-bfef-dafd3a848ec8", "id": 3} v 0) v1 -- 0x7f03c00b2c10 con 0x7f03c006daa0 2023-12-16T21:13:13.115 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.114+0000 7f03bcff9640 1 -- 172.21.15.121:0/2242665830 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5194ffa9-d0bb-4150-bfef-dafd3a848ec8", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f03b004d450 con 0x7f03c006daa0 2023-12-16T21:13:13.115 INFO:teuthology.orchestra.run.smithi121.stdout:3 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 msgr2=0x7f03c0159cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 0x7f03c0159cf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f03b002b940 tx=0x7f03b00046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 shutdown_connections 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c015e220 0x7f03c00aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03c013d060 0x7f03c00b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 --2- 172.21.15.121:0/2242665830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c006daa0 0x7f03c0159cf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 >> 172.21.15.121:0/2242665830 conn(0x7f03c0066270 msgr2=0x7f03c006b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.116 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 shutdown_connections 2023-12-16T21:13:13.117 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.115+0000 7f03c5ddc640 1 -- 172.21.15.121:0/2242665830 wait complete. 2023-12-16T21:13:13.126 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 8fdcefd0-2beb-469a-a9b1-52f12ebc98cf 4 2023-12-16T21:13:13.167 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.166+0000 7ff016433040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T21:13:13.179 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.179+0000 7f16ee04c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T21:13:13.241 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.241+0000 7ff016433040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T21:13:13.252 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.251+0000 7f16ee04c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T21:13:13.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.269+0000 7f01ae197640 1 Processor -- start 2023-12-16T21:13:13.269 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.269+0000 7f01ae197640 1 -- start start 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.269+0000 7f01ae197640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d19b0 0x7f01a80cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.269+0000 7f01ae197640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80d2e80 con 0x7f01a80d31c0 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.269+0000 7f01ae197640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80d28a0 con 0x7f01a80d1df0 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.269+0000 7f01ae197640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80cdc50 con 0x7f01a80d19b0 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a6ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d19b0 0x7f01a80cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f01a80d1df0 0x7f01a80b6470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:56774/0 (socket says 172.21.15.121:56774) 2023-12-16T21:13:13.270 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a7fff640 1 -- 172.21.15.121:0/374083711 learned_addr learned my addr 172.21.15.121:0/374083711 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-12-16T21:13:13.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a6ffd640 1 -- 172.21.15.121:0/374083711 >> v1:172.21.15.121:6789/0 conn(0x7f01a80d1df0 legacy=0x7f01a80b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a67fc640 1 -- 172.21.15.121:0/374083711 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568236766 0 0) 0x7f01a80d28a0 con 0x7f01a80d1df0 2023-12-16T21:13:13.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a6ffd640 1 -- 172.21.15.121:0/374083711 >> v1:172.21.15.46:6789/0 conn(0x7f01a80d31c0 legacy=0x7f01a80d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.271 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.270+0000 7f01a6ffd640 1 -- 172.21.15.121:0/374083711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80d2b10 con 0x7f01a80d19b0 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.271+0000 7f01a6ffd640 1 --2- 172.21.15.121:0/374083711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d19b0 0x7f01a80cdf30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f019c0097d0 tx=0x7f019c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2104447b3a4761b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.271+0000 7f01a67fc640 1 -- 172.21.15.121:0/374083711 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c031890 con 0x7f01a80d19b0 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.271+0000 7f01a67fc640 1 -- 172.21.15.121:0/374083711 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c0319f0 con 0x7f01a80d19b0 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.271+0000 7f01a67fc640 1 -- 172.21.15.121:0/374083711 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c031d80 con 0x7f01a80d19b0 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.271+0000 7f01ae197640 1 -- 172.21.15.121:0/374083711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d19b0 msgr2=0x7f01a80cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.271+0000 7f01ae197640 1 --2- 172.21.15.121:0/374083711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d19b0 0x7f01a80cdf30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f019c0097d0 tx=0x7f019c02b930 comp rx=0 tx=0).stop 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.272+0000 7f01ae197640 1 -- 172.21.15.121:0/374083711 shutdown_connections 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.272+0000 7f01ae197640 1 --2- 172.21.15.121:0/374083711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d19b0 0x7f01a80cdf30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.272 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.272+0000 7f01ae197640 1 -- 172.21.15.121:0/374083711 >> 172.21.15.121:0/374083711 conn(0x7f01a8072020 msgr2=0x7f01a8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.273 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.272+0000 7f01ae197640 1 -- 172.21.15.121:0/374083711 shutdown_connections 2023-12-16T21:13:13.273 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.272+0000 7f01ae197640 1 -- 172.21.15.121:0/374083711 wait complete. 2023-12-16T21:13:13.273 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.273+0000 7f01ae197640 1 Processor -- start 2023-12-16T21:13:13.274 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.274+0000 7f01ae197640 1 -- start start 2023-12-16T21:13:13.276 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01ae197640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01a80d19b0 0x7f01a81e4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01ae197640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80d1df0 0x7f01a814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01ae197640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d31c0 0x7f01a81e30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01ae197640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a815ba70 con 0x7f01a80d1df0 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01ae197640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a81e3c30 con 0x7f01a80d19b0 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01ae197640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a81e3f10 con 0x7f01a80d31c0 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01a80d19b0 0x7f01a81e4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d31c0 0x7f01a81e30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a77fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01a80d19b0 0x7f01a81e4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47622/0 (socket says 172.21.15.121:47622) 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a77fe640 1 -- 172.21.15.121:0/3005605631 learned_addr learned my addr 172.21.15.121:0/3005605631 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a6ffd640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80d1df0 0x7f01a814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a7fff640 1 -- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01a80d19b0 msgr2=0x7f01a81e4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a7fff640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01a80d19b0 0x7f01a81e4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a7fff640 1 -- 172.21.15.121:0/3005605631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80d1df0 msgr2=0x7f01a814e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.277 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a7fff640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80d1df0 0x7f01a814e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.275+0000 7f01a7fff640 1 -- 172.21.15.121:0/3005605631 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c009400 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f01a7fff640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d31c0 0x7f01a81e30c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f01a0001560 tx=0x7f01a000cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a000ce30 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a000a640 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a81e41f0 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a000aa10 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a81e47b0 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f01a00070d0 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.276+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f01a0012910 con 0x7f01a80d31c0 2023-12-16T21:13:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.277+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a8069400 con 0x7f01a80d31c0 2023-12-16T21:13:13.279 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.279+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f01a0018070 con 0x7f01a80d31c0 2023-12-16T21:13:13.292 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.291+0000 7ff016433040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T21:13:13.301 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.300+0000 7f16ee04c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T21:13:13.345 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.345+0000 7ff016433040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T21:13:13.352 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.352+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8fdcefd0-2beb-469a-a9b1-52f12ebc98cf", "id": 4} v 0) v1 -- 0x7f01a81e49a0 con 0x7f01a80d31c0 2023-12-16T21:13:13.355 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.354+0000 7f16ee04c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T21:13:13.355 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.355+0000 7f018ffff640 1 -- 172.21.15.121:0/3005605631 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8fdcefd0-2beb-469a-a9b1-52f12ebc98cf", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f01a0012c90 con 0x7f01a80d31c0 2023-12-16T21:13:13.355 INFO:teuthology.orchestra.run.smithi121.stdout:4 2023-12-16T21:13:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d31c0 msgr2=0x7f01a81e30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d31c0 0x7f01a81e30c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f01a0001560 tx=0x7f01a000cbe0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 shutdown_connections 2023-12-16T21:13:13.356 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a80d1df0 0x7f01a814e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01a80d31c0 0x7f01a81e30c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 --2- 172.21.15.121:0/3005605631 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01a80d19b0 0x7f01a81e4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 >> 172.21.15.121:0/3005605631 conn(0x7f01a8072020 msgr2=0x7f01a815c840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 shutdown_connections 2023-12-16T21:13:13.357 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.356+0000 7f01ae197640 1 -- 172.21.15.121:0/3005605631 wait complete. 2023-12-16T21:13:13.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new aa80e233-3e84-470b-8c10-bb8abe6f3058 5 2023-12-16T21:13:13.396 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.396+0000 7ff016433040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T21:13:13.404 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.404+0000 7f16ee04c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T21:13:13.467 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.467+0000 7fa78e009640 1 Processor -- start 2023-12-16T21:13:13.468 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.467+0000 7fa78e009640 1 -- start start 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa78e009640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa788143680 0x7fa78815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa78e009640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78806e960 con 0x7fa78806abe0 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa78e009640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78815dec0 con 0x7fa7881432a0 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa78e009640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa78815e1a0 con 0x7fa788143680 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa787fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa788143680 0x7fa78815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa7877fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa7881432a0 0x7fa78806a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:56778/0 (socket says 172.21.15.121:56778) 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa7877fe640 1 -- 172.21.15.121:0/371092363 learned_addr learned my addr 172.21.15.121:0/371092363 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa7867fc640 1 -- 172.21.15.121:0/371092363 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836840159 0 0) 0x7fa78815dec0 con 0x7fa7881432a0 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa787fff640 1 -- 172.21.15.121:0/371092363 >> v1:172.21.15.121:6789/0 conn(0x7fa7881432a0 legacy=0x7fa78806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa787fff640 1 -- 172.21.15.121:0/371092363 >> v1:172.21.15.46:6789/0 conn(0x7fa78806abe0 legacy=0x7fa78806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.469 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.468+0000 7fa787fff640 1 -- 172.21.15.121:0/371092363 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78815e480 con 0x7fa788143680 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.469+0000 7fa787fff640 1 --2- 172.21.15.121:0/371092363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa788143680 0x7fa78815d8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa780009dc0 tx=0x7fa78002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6190e8ad5b1cb19 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.469+0000 7fa7867fc640 1 -- 172.21.15.121:0/371092363 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78002ccd0 con 0x7fa788143680 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.469+0000 7fa7867fc640 1 -- 172.21.15.121:0/371092363 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78002ce30 con 0x7fa788143680 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.469+0000 7fa7867fc640 1 -- 172.21.15.121:0/371092363 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780034660 con 0x7fa788143680 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 -- 172.21.15.121:0/371092363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa788143680 msgr2=0x7fa78815d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 --2- 172.21.15.121:0/371092363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa788143680 0x7fa78815d8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa780009dc0 tx=0x7fa78002bf20 comp rx=0 tx=0).stop 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 -- 172.21.15.121:0/371092363 shutdown_connections 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 --2- 172.21.15.121:0/371092363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa788143680 0x7fa78815d8e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 -- 172.21.15.121:0/371092363 >> 172.21.15.121:0/371092363 conn(0x7fa7880b1de0 msgr2=0x7fa7880b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 -- 172.21.15.121:0/371092363 shutdown_connections 2023-12-16T21:13:13.471 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.470+0000 7fa78e009640 1 -- 172.21.15.121:0/371092363 wait complete. 2023-12-16T21:13:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.471+0000 7fa78e009640 1 Processor -- start 2023-12-16T21:13:13.472 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.471+0000 7fa78e009640 1 -- start start 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa78e009640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa78806abe0 0x7fa78813c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa78e009640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 0x7fa7880aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa78e009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa788143680 0x7fa7880af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa78e009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa788142bd0 con 0x7fa788143680 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa78e009640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78813f570 con 0x7fa7881432a0 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa78e009640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7881f6360 con 0x7fa78806abe0 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 0x7fa7880aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa7877fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa78806abe0 0x7fa78813c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa786ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 0x7fa7880aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47638/0 (socket says 172.21.15.121:47638) 2023-12-16T21:13:13.474 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa786ffd640 1 -- 172.21.15.121:0/471359725 learned_addr learned my addr 172.21.15.121:0/471359725 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.472+0000 7fa787fff640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa788143680 0x7fa7880af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa7877fe640 1 -- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 msgr2=0x7fa7880aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa7877fe640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 0x7fa7880aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa7877fe640 1 -- 172.21.15.121:0/471359725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa788143680 msgr2=0x7fa7880af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa7877fe640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa788143680 0x7fa7880af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa7877fe640 1 -- 172.21.15.121:0/471359725 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa780009a70 con 0x7fa78806abe0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa786ffd640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 0x7fa7880aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa787fff640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa788143680 0x7fa7880af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa7877fe640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa78806abe0 0x7fa78813c9e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c410 tx=0x7fa77800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7780074e0 con 0x7fa78806abe0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa778007660 con 0x7fa78806abe0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7881f70e0 con 0x7fa78806abe0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa778004a20 con 0x7fa78806abe0 2023-12-16T21:13:13.475 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.473+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7881f68e0 con 0x7fa78806abe0 2023-12-16T21:13:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.474+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa77801d3f0 con 0x7fa78806abe0 2023-12-16T21:13:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.474+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fa778013940 con 0x7fa78806abe0 2023-12-16T21:13:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.474+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa788067420 con 0x7fa78806abe0 2023-12-16T21:13:13.476 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.476+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7fa778013ba0 con 0x7fa78806abe0 2023-12-16T21:13:13.549 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.549+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "aa80e233-3e84-470b-8c10-bb8abe6f3058", "id": 5} v 0) v1 -- 0x7fa788161bb0 con 0x7fa78806abe0 2023-12-16T21:13:13.552 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.552+0000 7fa76ffff640 1 -- 172.21.15.121:0/471359725 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aa80e233-3e84-470b-8c10-bb8abe6f3058", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa778013e10 con 0x7fa78806abe0 2023-12-16T21:13:13.552 INFO:teuthology.orchestra.run.smithi121.stdout:5 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa78806abe0 msgr2=0x7fa78813c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa78806abe0 0x7fa78813c9e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c410 tx=0x7fa77800c8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 shutdown_connections 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa788143680 0x7fa7880af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7881432a0 0x7fa7880aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 --2- 172.21.15.121:0/471359725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa78806abe0 0x7fa78813c9e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 >> 172.21.15.121:0/471359725 conn(0x7fa7880b1de0 msgr2=0x7fa7880b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 shutdown_connections 2023-12-16T21:13:13.554 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.553+0000 7fa78e009640 1 -- 172.21.15.121:0/471359725 wait complete. 2023-12-16T21:13:13.570 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 2684c217-04a6-4b3e-9f3d-3e69c41432d3 6 2023-12-16T21:13:13.714 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.711+0000 7f8fa0f4c640 1 Processor -- start 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.711+0000 7f8fa0f4c640 1 -- start start 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.712+0000 7f8fa0f4c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8f9c0d31c0 0x7f8f9c0d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.712+0000 7f8fa0f4c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8f9c0d19b0 0x7f8f9c0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.712+0000 7f8fa0f4c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9c0d2e80 con 0x7f8f9c0d1e20 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.712+0000 7f8fa0f4c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9c0d28a0 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.712+0000 7f8fa0f4c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f9c0d3ae0 con 0x7f8f9c0d19b0 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8f9c0d19b0 0x7f8f9c0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8f9c0d19b0 0x7f8f9c0cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47646/0 (socket says 172.21.15.121:47646) 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 -- 172.21.15.121:0/901017543 learned_addr learned my addr 172.21.15.121:0/901017543 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f9a59c640 1 --2- 172.21.15.121:0/901017543 >> v2:172.21.15.46:3300/0 conn(0x7f8f9c0d31c0 0x7f8f9c0d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 -- 172.21.15.121:0/901017543 >> v2:172.21.15.46:3300/0 conn(0x7f8f9c0d31c0 msgr2=0x7f8f9c0d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:13.715 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 --2- 172.21.15.121:0/901017543 >> v2:172.21.15.46:3300/0 conn(0x7f8f9c0d31c0 0x7f8f9c0d35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 -- 172.21.15.121:0/901017543 >> v1:172.21.15.46:6789/0 conn(0x7f8f9c0d1e20 legacy=0x7f8f9c0b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f99d9b640 1 -- 172.21.15.121:0/901017543 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c0cdc50 con 0x7f8f9c0d19b0 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.713+0000 7f8f9a59c640 1 --2- 172.21.15.121:0/901017543 >> v2:172.21.15.46:3300/0 conn(0x7f8f9c0d31c0 0x7f8f9c0d35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.714+0000 7f8f99d9b640 1 --2- 172.21.15.121:0/901017543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d19b0 0x7f8f9c0cdf30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c06bf30 tx=0x7f8f9002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3713089ba8e90ce3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.714+0000 7f8f9959a640 1 -- 172.21.15.121:0/901017543 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90030860 con 0x7f8f9c0d19b0 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.714+0000 7f8f9959a640 1 -- 172.21.15.121:0/901017543 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f900309c0 con 0x7f8f9c0d19b0 2023-12-16T21:13:13.716 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.714+0000 7f8f9959a640 1 -- 172.21.15.121:0/901017543 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90030d50 con 0x7f8f9c0d19b0 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/901017543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d19b0 msgr2=0x7f8f9c0cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/901017543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d19b0 0x7f8f9c0cdf30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c06bf30 tx=0x7f8f9002b9a0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/901017543 shutdown_connections 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/901017543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d19b0 0x7f8f9c0cdf30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/901017543 >> v2:172.21.15.46:3300/0 conn(0x7f8f9c0d31c0 0x7f8f9c0d35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/901017543 >> 172.21.15.121:0/901017543 conn(0x7f8f9c072020 msgr2=0x7f8f9c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/901017543 shutdown_connections 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.715+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/901017543 wait complete. 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.716+0000 7f8fa0f4c640 1 Processor -- start 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.716+0000 7f8fa0f4c640 1 -- start start 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8fa0f4c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f9c0d19b0 0x7f8f9c1f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.719 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8fa0f4c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c0d1e20 0x7f8f9c150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8fa0f4c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 0x7f8f9c154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8fa0f4c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c1f3fe0 con 0x7f8f9c0d1e20 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8fa0f4c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c0b3d70 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8fa0f4c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c0b4050 con 0x7f8f9c0d19b0 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8f9ad9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 0x7f8f9c154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8f9a59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f9c0d19b0 0x7f8f9c1f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8f9ad9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 0x7f8f9c154cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47656/0 (socket says 172.21.15.121:47656) 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8f9ad9d640 1 -- 172.21.15.121:0/3420200472 learned_addr learned my addr 172.21.15.121:0/3420200472 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.717+0000 7f8f99d9b640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c0d1e20 0x7f8f9c150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9ad9d640 1 -- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f9c0d19b0 msgr2=0x7f8f9c1f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9ad9d640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f9c0d19b0 0x7f8f9c1f3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9ad9d640 1 -- 172.21.15.121:0/3420200472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c0d1e20 msgr2=0x7f8f9c150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9ad9d640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c0d1e20 0x7f8f9c150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9ad9d640 1 -- 172.21.15.121:0/3420200472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900094d0 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9a59c640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f9c0d19b0 0x7f8f9c1f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:13.720 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f9ad9d640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 0x7f8f9c154cf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400c680 tx=0x7f8f9400cb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94003b70 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f94003cd0 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c0b4330 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94005310 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c0b48f0 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8f9400a640 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.718+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f8f94012a60 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.721 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.719+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c072730 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.722 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.721+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f8f94012c70 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.795 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.795+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2684c217-04a6-4b3e-9f3d-3e69c41432d3", "id": 6} v 0) v1 -- 0x7f8f9c069400 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.798 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.798+0000 7f8f8affd640 1 -- 172.21.15.121:0/3420200472 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2684c217-04a6-4b3e-9f3d-3e69c41432d3", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f94012e10 con 0x7f8f9c0d31c0 2023-12-16T21:13:13.798 INFO:teuthology.orchestra.run.smithi121.stdout:6 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 msgr2=0x7f8f9c154cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 0x7f8f9c154cf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400c680 tx=0x7f8f9400cb40 comp rx=0 tx=0).stop 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 shutdown_connections 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c0d1e20 0x7f8f9c150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f9c0d31c0 0x7f8f9c154cf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 --2- 172.21.15.121:0/3420200472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f9c0d19b0 0x7f8f9c1f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 >> 172.21.15.121:0/3420200472 conn(0x7f8f9c072020 msgr2=0x7f8f9c0cb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 shutdown_connections 2023-12-16T21:13:13.800 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.799+0000 7f8fa0f4c640 1 -- 172.21.15.121:0/3420200472 wait complete. 2023-12-16T21:13:13.802 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.802+0000 7ff016433040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T21:13:13.805 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.805+0000 7f16ee04c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T21:13:13.810 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 4c383297-1347-41c9-92a3-5cb522618343 7 2023-12-16T21:13:13.910 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.910+0000 7f451f9d7640 1 Processor -- start 2023-12-16T21:13:13.911 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.910+0000 7f451f9d7640 1 -- start start 2023-12-16T21:13:13.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451f9d7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4518144c60 0x7f451806a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451f9d7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f451806f370 0x7f451806aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451f9d7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451806f8e0 con 0x7f45180ccac0 2023-12-16T21:13:13.912 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451f9d7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45180cda30 con 0x7f4518144c60 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451f9d7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45180cdd10 con 0x7f451806f370 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451cf72640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f451806f370 0x7f451806aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451cf72640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f451806f370 0x7f451806aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:47660/0 (socket says 172.21.15.121:47660) 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451cf72640 1 -- 172.21.15.121:0/452688794 learned_addr learned my addr 172.21.15.121:0/452688794 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.911+0000 7f451d773640 1 --2- 172.21.15.121:0/452688794 >> v2:172.21.15.46:3300/0 conn(0x7f4518144c60 0x7f451806a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f451cf72640 1 -- 172.21.15.121:0/452688794 >> v2:172.21.15.46:3300/0 conn(0x7f4518144c60 msgr2=0x7f451806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f451cf72640 1 --2- 172.21.15.121:0/452688794 >> v2:172.21.15.46:3300/0 conn(0x7f4518144c60 0x7f451806a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f451cf72640 1 -- 172.21.15.121:0/452688794 >> v1:172.21.15.46:6789/0 conn(0x7f45180ccac0 legacy=0x7f451815ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f451cf72640 1 -- 172.21.15.121:0/452688794 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45180cdff0 con 0x7f451806f370 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f451d773640 1 --2- 172.21.15.121:0/452688794 >> v2:172.21.15.46:3300/0 conn(0x7f4518144c60 0x7f451806a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:13.913 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f451cf72640 1 --2- 172.21.15.121:0/452688794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f451806f370 0x7f451806aa10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f451806cee0 tx=0x7f450802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71d95d1d237c7e68 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.914 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f450ffff640 1 -- 172.21.15.121:0/452688794 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508031870 con 0x7f451806f370 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f450ffff640 1 -- 172.21.15.121:0/452688794 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45080319d0 con 0x7f451806f370 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.912+0000 7f450ffff640 1 -- 172.21.15.121:0/452688794 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4508031d60 con 0x7f451806f370 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 -- 172.21.15.121:0/452688794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f451806f370 msgr2=0x7f451806aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 --2- 172.21.15.121:0/452688794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f451806f370 0x7f451806aa10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f451806cee0 tx=0x7f450802b910 comp rx=0 tx=0).stop 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 -- 172.21.15.121:0/452688794 shutdown_connections 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 --2- 172.21.15.121:0/452688794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f451806f370 0x7f451806aa10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 --2- 172.21.15.121:0/452688794 >> v2:172.21.15.46:3300/0 conn(0x7f4518144c60 0x7f451806a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 -- 172.21.15.121:0/452688794 >> 172.21.15.121:0/452688794 conn(0x7f45180b1de0 msgr2=0x7f45180b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:13.917 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 -- 172.21.15.121:0/452688794 shutdown_connections 2023-12-16T21:13:13.918 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.913+0000 7f451f9d7640 1 -- 172.21.15.121:0/452688794 wait complete. 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.914+0000 7f451f9d7640 1 Processor -- start 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.914+0000 7f451f9d7640 1 -- start start 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451f9d7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 0x7f45180b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451f9d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45180ccac0 0x7f45180b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451f9d7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4518144c60 0x7f451813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451f9d7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451813f3d0 con 0x7f45180ccac0 2023-12-16T21:13:13.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451f9d7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45180b3d20 con 0x7f4518144c60 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451f9d7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45180b4000 con 0x7f451806f370 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451d773640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 0x7f45180b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451df74640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4518144c60 0x7f451813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451d773640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 0x7f45180b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44588/0 (socket says 172.21.15.121:44588) 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451d773640 1 -- 172.21.15.121:0/1573659870 learned_addr learned my addr 172.21.15.121:0/1573659870 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.915+0000 7f451cf72640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45180ccac0 0x7f45180b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451d773640 1 -- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4518144c60 msgr2=0x7f451813ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451d773640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4518144c60 0x7f451813ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451d773640 1 -- 172.21.15.121:0/1573659870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45180ccac0 msgr2=0x7f45180b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451d773640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45180ccac0 0x7f45180b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451d773640 1 -- 172.21.15.121:0/1573659870 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45080094d0 con 0x7f451806f370 2023-12-16T21:13:13.921 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451df74640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4518144c60 0x7f451813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451d773640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 0x7f45180b4ae0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f451400d7d0 tx=0x7f451400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514004030 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4514004190 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45180b42e0 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514004520 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f451400b9c0 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4518141b70 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.916+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f4514003d40 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.917+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4518073ba0 con 0x7f451806f370 2023-12-16T21:13:13.922 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.921+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f451400be60 con 0x7f451806f370 2023-12-16T21:13:13.946 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:13.945+0000 7ff016433040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T21:13:13.947 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:13.947+0000 7f16ee04c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T21:13:13.993 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.992+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4c383297-1347-41c9-92a3-5cb522618343", "id": 7} v 0) v1 -- 0x7f4518067420 con 0x7f451806f370 2023-12-16T21:13:13.999 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:13.999+0000 7f450dffb640 1 -- 172.21.15.121:0/1573659870 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4c383297-1347-41c9-92a3-5cb522618343", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4514003610 con 0x7f451806f370 2023-12-16T21:13:13.999 INFO:teuthology.orchestra.run.smithi121.stdout:7 2023-12-16T21:13:14.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 msgr2=0x7f45180b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 0x7f45180b4ae0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f451400d7d0 tx=0x7f451400dc90 comp rx=0 tx=0).stop 2023-12-16T21:13:14.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 shutdown_connections 2023-12-16T21:13:14.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45180ccac0 0x7f45180b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4518144c60 0x7f451813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.000 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 --2- 172.21.15.121:0/1573659870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f451806f370 0x7f45180b4ae0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 >> 172.21.15.121:0/1573659870 conn(0x7f45180b1de0 msgr2=0x7f45180b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:14.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 shutdown_connections 2023-12-16T21:13:14.001 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:14.000+0000 7f451f9d7640 1 -- 172.21.15.121:0/1573659870 wait complete. 2023-12-16T21:13:14.010 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-16T21:13:14.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-16T21:13:14.012 INFO:tasks.ceph.osd.0:Started 2023-12-16T21:13:14.012 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-16T21:13:14.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-16T21:13:14.015 INFO:tasks.ceph.osd.1:Started 2023-12-16T21:13:14.015 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-16T21:13:14.015 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-16T21:13:14.017 INFO:tasks.ceph.osd.2:Started 2023-12-16T21:13:14.018 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-16T21:13:14.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-16T21:13:14.020 INFO:tasks.ceph.osd.3:Started 2023-12-16T21:13:14.020 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-16T21:13:14.020 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-16T21:13:14.023 INFO:tasks.ceph.osd.4:Started 2023-12-16T21:13:14.023 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-16T21:13:14.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-16T21:13:14.025 INFO:tasks.ceph.osd.5:Started 2023-12-16T21:13:14.025 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-16T21:13:14.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-16T21:13:14.027 INFO:tasks.ceph.osd.6:Started 2023-12-16T21:13:14.027 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-16T21:13:14.028 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-16T21:13:14.030 INFO:tasks.ceph.osd.7:Started 2023-12-16T21:13:14.031 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:14.102 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.101+0000 7f16ee04c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T21:13:14.114 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.114+0000 7ff016433040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T21:13:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.160+0000 7f5f62c15640 1 Processor -- start 2023-12-16T21:13:14.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.160+0000 7f5f62c15640 1 -- start start 2023-12-16T21:13:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f62c15640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5f5c15b130 0x7f5f5c0d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:14.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f62c15640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c0b3f10 con 0x7f5f5c0cf320 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f62c15640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c0d0a50 con 0x7f5f5c15f760 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f62c15640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5c15fd20 con 0x7f5f5c15b130 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f611b2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5f5c0cf320 0x7f5f5c0cd310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57004/0 (socket says 172.21.15.46:57004) 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f611b2640 1 -- 172.21.15.46:0/2314107894 learned_addr learned my addr 172.21.15.46:0/2314107894 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.161+0000 7f5f609b1640 1 --2- 172.21.15.46:0/2314107894 >> v2:172.21.15.46:3300/0 conn(0x7f5f5c15b130 0x7f5f5c0d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.162+0000 7f5f609b1640 1 -- 172.21.15.46:0/2314107894 >> v1:172.21.15.121:6789/0 conn(0x7f5f5c15f760 legacy=0x7f5f5c0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.162+0000 7f5f609b1640 1 -- 172.21.15.46:0/2314107894 >> v1:172.21.15.46:6789/0 conn(0x7f5f5c0cf320 legacy=0x7f5f5c0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.162+0000 7f5f609b1640 1 -- 172.21.15.46:0/2314107894 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c0d0d30 con 0x7f5f5c15b130 2023-12-16T21:13:14.163 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.162+0000 7f16ee04c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T21:13:14.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.162+0000 7f5f609b1640 1 --2- 172.21.15.46:0/2314107894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0d01f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5f500097a0 tx=0x7f5f5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbc2aeb165d6a61 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:14.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/2314107894 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50031830 con 0x7f5f5c15b130 2023-12-16T21:13:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/2314107894 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f50031990 con 0x7f5f5c15b130 2023-12-16T21:13:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f5b7fe640 1 -- 172.21.15.46:0/2314107894 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50031d20 con 0x7f5f5c15b130 2023-12-16T21:13:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 -- 172.21.15.46:0/2314107894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 msgr2=0x7f5f5c0d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 --2- 172.21.15.46:0/2314107894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0d01f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5f500097a0 tx=0x7f5f5002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 -- 172.21.15.46:0/2314107894 shutdown_connections 2023-12-16T21:13:14.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 --2- 172.21.15.46:0/2314107894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0d01f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 -- 172.21.15.46:0/2314107894 >> 172.21.15.46:0/2314107894 conn(0x7f5f5c0b2c10 msgr2=0x7f5f5c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 -- 172.21.15.46:0/2314107894 shutdown_connections 2023-12-16T21:13:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.163+0000 7f5f62c15640 1 -- 172.21.15.46:0/2314107894 wait complete. 2023-12-16T21:13:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.164+0000 7f5f62c15640 1 Processor -- start 2023-12-16T21:13:14.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.164+0000 7f5f62c15640 1 -- start start 2023-12-16T21:13:14.168 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.166+0000 7ff016433040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.165+0000 7f5f62c15640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 0x7f5f5c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.165+0000 7f5f62c15640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.165+0000 7f5f62c15640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f5c15f760 0x7f5f5c0c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f62c15640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c0c51f0 con 0x7f5f5c15b130 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f609b1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 0x7f5f5c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f62c15640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c0c54d0 con 0x7f5f5c0cf320 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f609b1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 0x7f5f5c0c49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:37364/0 (socket says 172.21.15.46:37364) 2023-12-16T21:13:14.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f609b1640 1 -- 172.21.15.46:0/1086326005 learned_addr learned my addr 172.21.15.46:0/1086326005 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5c0c57b0 con 0x7f5f5c15f760 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f611b2640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f5c15f760 0x7f5f5c0c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 -- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f5c15f760 msgr2=0x7f5f5c0c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f5c15f760 0x7f5f5c0c6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 -- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 msgr2=0x7f5f5c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 0x7f5f5c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 -- 172.21.15.46:0/1086326005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50009400 con 0x7f5f5c15b130 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f609b1640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 0x7f5f5c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f5bfff640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0b5bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c390 tx=0x7f5f4800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f480052d0 con 0x7f5f5c15b130 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f48005430 con 0x7f5f5c15b130 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c0c5a90 con 0x7f5f5c15b130 2023-12-16T21:13:14.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48003830 con 0x7f5f5c15b130 2023-12-16T21:13:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.166+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c0c4ee0 con 0x7f5f5c15b130 2023-12-16T21:13:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.167+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5f48003c90 con 0x7f5f5c15b130 2023-12-16T21:13:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.167+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5f48011e00 con 0x7f5f5c15b130 2023-12-16T21:13:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.168+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5c067fc0 con 0x7f5f5c15b130 2023-12-16T21:13:14.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.170+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f5f48017070 con 0x7f5f5c15b130 2023-12-16T21:13:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.232+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f5c067420 con 0x7f5f5c15b130 2023-12-16T21:13:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.232+0000 7f5f597fa640 1 -- 172.21.15.46:0/1086326005 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5f48017070 con 0x7f5f5c15b130 2023-12-16T21:13:14.233 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:14.233 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:13.995107+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 msgr2=0x7f5f5c0b5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0b5bf0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c390 tx=0x7f5f4800c850 comp rx=0 tx=0).stop 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 shutdown_connections 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f5c15b130 0x7f5f5c0b5bf0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f5c15f760 0x7f5f5c0c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 --2- 172.21.15.46:0/1086326005 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f5c0cf320 0x7f5f5c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 >> 172.21.15.46:0/1086326005 conn(0x7f5f5c0b2c10 msgr2=0x7f5f5c073440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 shutdown_connections 2023-12-16T21:13:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:14.233+0000 7f5f62c15640 1 -- 172.21.15.46:0/1086326005 wait complete. 2023-12-16T21:13:14.244 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-16T21:13:14.245 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-16T21:13:14.266 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.266+0000 7ff016433040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:13:14.267 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.267+0000 7f16ee04c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:13:14.363 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.362+0000 7ff016433040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T21:13:14.366 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.366+0000 7f16ee04c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T21:13:14.513 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.513+0000 7f16ee04c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:13:14.516 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.516+0000 7ff016433040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T21:13:14.558 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.557+0000 7ff016433040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T21:13:14.558 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.557+0000 7f16ee04c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T21:13:14.636 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-12-16T21:13:14.635+0000 7f0be181d500 -1 Falling back to public interface 2023-12-16T21:13:14.636 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-12-16T21:13:14.636+0000 7f2b2aca5500 -1 Falling back to public interface 2023-12-16T21:13:14.644 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-12-16T21:13:14.644+0000 7f4f594d2500 -1 Falling back to public interface 2023-12-16T21:13:14.645 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-12-16T21:13:14.644+0000 7f319340c500 -1 Falling back to public interface 2023-12-16T21:13:14.645 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-12-16T21:13:14.645+0000 7fb075dbc500 -1 Falling back to public interface 2023-12-16T21:13:14.646 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-12-16T21:13:14.646+0000 7f4f58479640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6815/0: (98) Address already in use 2023-12-16T21:13:14.650 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-12-16T21:13:14.650+0000 7ff01bec0500 -1 Falling back to public interface 2023-12-16T21:13:14.653 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-12-16T21:13:14.652+0000 7f5dca69e500 -1 Falling back to public interface 2023-12-16T21:13:14.662 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-12-16T21:13:14.662+0000 7fefe90c9500 -1 Falling back to public interface 2023-12-16T21:13:14.700 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:14.700+0000 7ff016433040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T21:13:14.709 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:14.708+0000 7f16ee04c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T21:13:15.050 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.142+0000 7f2f7800e640 1 Processor -- start 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.143+0000 7f2f7800e640 1 -- start start 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.143+0000 7f2f7800e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2f700b0510 0x7f2f700cce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.143+0000 7f2f7800e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f700cedc0 0x7f2f70099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.143+0000 7f2f7800e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f700d01b0 con 0x7f2f70099fb0 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.143+0000 7f2f7800e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f7009a440 con 0x7f2f700b0510 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.143+0000 7f2f7800e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f700b08f0 con 0x7f2f700cedc0 2023-12-16T21:13:15.148 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f765ab640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2f70099fb0 0x7f2f7015cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:57014/0 (socket says 172.21.15.46:57014) 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f765ab640 1 -- 172.21.15.46:0/2255542902 learned_addr learned my addr 172.21.15.46:0/2255542902 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f75daa640 1 --2- 172.21.15.46:0/2255542902 >> v2:172.21.15.121:3300/0 conn(0x7f2f700b0510 0x7f2f700cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f755a9640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f700cedc0 0x7f2f70099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f74da8640 1 -- 172.21.15.46:0/2255542902 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692525105 0 0) 0x7f2f700d01b0 con 0x7f2f70099fb0 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f74da8640 1 -- 172.21.15.46:0/2255542902 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c003520 con 0x7f2f70099fb0 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f75daa640 1 -- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f700cedc0 msgr2=0x7f2f70099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f75daa640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f700cedc0 0x7f2f70099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f75daa640 1 -- 172.21.15.46:0/2255542902 >> v1:172.21.15.46:6789/0 conn(0x7f2f70099fb0 legacy=0x7f2f7015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f75daa640 1 -- 172.21.15.46:0/2255542902 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700cf5a0 con 0x7f2f700b0510 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.144+0000 7f2f755a9640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f700cedc0 0x7f2f70099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.145+0000 7f2f75daa640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700b0510 0x7f2f700cce50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2f640097a0 tx=0x7f2f6402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4df85b3e207c707f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.145+0000 7f2f74da8640 1 -- 172.21.15.46:0/2255542902 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64031830 con 0x7f2f700b0510 2023-12-16T21:13:15.149 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.145+0000 7f2f74da8640 1 -- 172.21.15.46:0/2255542902 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f64031990 con 0x7f2f700b0510 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.145+0000 7f2f74da8640 1 -- 172.21.15.46:0/2255542902 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f64031d20 con 0x7f2f700b0510 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 -- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700b0510 msgr2=0x7f2f700cce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700b0510 0x7f2f700cce50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2f640097a0 tx=0x7f2f6402b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 -- 172.21.15.46:0/2255542902 shutdown_connections 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f700cedc0 0x7f2f70099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 --2- 172.21.15.46:0/2255542902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700b0510 0x7f2f700cce50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 -- 172.21.15.46:0/2255542902 >> 172.21.15.46:0/2255542902 conn(0x7f2f700b2b50 msgr2=0x7f2f700b2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 -- 172.21.15.46:0/2255542902 shutdown_connections 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.146+0000 7f2f7800e640 1 -- 172.21.15.46:0/2255542902 wait complete. 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.147+0000 7f2f7800e640 1 Processor -- start 2023-12-16T21:13:15.150 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.147+0000 7f2f7800e640 1 -- start start 2023-12-16T21:13:15.152 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f7800e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f70099fb0 0x7f2f7006ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:15.152 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f7800e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 0x7f2f7015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:15.152 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f7800e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700cedc0 0x7f2f7015c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:15.152 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f7800e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7006e830 con 0x7f2f700b0510 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f7800e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7006eb10 con 0x7f2f700cedc0 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f7800e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f7006edf0 con 0x7f2f70099fb0 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 0x7f2f7015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 0x7f2f7015c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44162/0 (socket says 172.21.15.46:44162) 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 -- 172.21.15.46:0/3509797962 learned_addr learned my addr 172.21.15.46:0/3509797962 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f75daa640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f70099fb0 0x7f2f7006ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f765ab640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700cedc0 0x7f2f7015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 -- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f70099fb0 msgr2=0x7f2f7006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f70099fb0 0x7f2f7006ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 -- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700cedc0 msgr2=0x7f2f7015c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700cedc0 0x7f2f7015c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f755a9640 1 -- 172.21.15.46:0/3509797962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c009400 con 0x7f2f700b0510 2023-12-16T21:13:15.153 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f75daa640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f70099fb0 0x7f2f7006ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.148+0000 7f2f765ab640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700cedc0 0x7f2f7015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f755a9640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 0x7f2f7015c1d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00e960 tx=0x7f2f6c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c00ca00 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f6c00cb60 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f64009400 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c013470 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7006f3b0 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2f6c005c90 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.149+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2f6c013ca0 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.150+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70067420 con 0x7f2f700b0510 2023-12-16T21:13:15.154 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.152+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72915 (secure 0 0 0) 0x7f2f6c021060 con 0x7f2f700b0510 2023-12-16T21:13:15.214 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.214+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f700c5c00 con 0x7f2f700b0510 2023-12-16T21:13:15.215 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.214+0000 7f2f627fc640 1 -- 172.21.15.46:0/3509797962 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2f6c017070 con 0x7f2f700b0510 2023-12-16T21:13:15.215 INFO:teuthology.misc.health.smithi046.stdout: 2023-12-16T21:13:15.215 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":10,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:13.995107+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 msgr2=0x7f2f7015c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 0x7f2f7015c1d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00e960 tx=0x7f2f6c00ee20 comp rx=0 tx=0).stop 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 shutdown_connections 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f700b0510 0x7f2f7015c1d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f700cedc0 0x7f2f7015c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 --2- 172.21.15.46:0/3509797962 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f70099fb0 0x7f2f7006ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 >> 172.21.15.46:0/3509797962 conn(0x7f2f700b2b50 msgr2=0x7f2f701602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 shutdown_connections 2023-12-16T21:13:15.216 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:15.215+0000 7f2f7800e640 1 -- 172.21.15.46:0/3509797962 wait complete. 2023-12-16T21:13:15.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-16T21:13:15.507 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:15.506+0000 7ff016433040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T21:13:15.554 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:15.553+0000 7f16ee04c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T21:13:15.780 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:15.779+0000 7ff016433040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T21:13:15.830 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:15.830+0000 7ff016433040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T21:13:15.842 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:15.842+0000 7f16ee04c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T21:13:15.896 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:15.895+0000 7f16ee04c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T21:13:15.929 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:15.929+0000 7ff016433040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T21:13:16.006 INFO:tasks.ceph.mgr.y.smithi046.stderr:2023-12-16T21:13:16.005+0000 7f16ee04c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T21:13:16.955 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:16.955+0000 7fefb6efa640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-16T21:13:16.956 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-12-16T21:13:16.956+0000 7fefb6efa640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-16T21:13:18.098 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-12-16T21:13:18.098+0000 7f0be181d500 -1 osd.0 0 log_to_monitors true 2023-12-16T21:13:18.136 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-12-16T21:13:18.136+0000 7f4f594d2500 -1 osd.5 0 log_to_monitors true 2023-12-16T21:13:18.150 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-12-16T21:13:18.150+0000 7f319340c500 -1 osd.1 0 log_to_monitors true 2023-12-16T21:13:18.150 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-12-16T21:13:18.150+0000 7f5dca69e500 -1 osd.2 0 log_to_monitors true 2023-12-16T21:13:18.173 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-12-16T21:13:18.173+0000 7f2b2aca5500 -1 osd.7 0 log_to_monitors true 2023-12-16T21:13:18.176 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-12-16T21:13:18.175+0000 7ff01bec0500 -1 osd.4 0 log_to_monitors true 2023-12-16T21:13:18.177 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-12-16T21:13:18.177+0000 7fefe90c9500 -1 osd.3 0 log_to_monitors true 2023-12-16T21:13:18.188 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-12-16T21:13:18.188+0000 7fb075dbc500 -1 osd.6 0 log_to_monitors true 2023-12-16T21:13:21.437 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-12-16T21:13:21.437+0000 7fefe4836640 -1 osd.3 0 waiting for initial osdmap 2023-12-16T21:13:21.453 INFO:tasks.ceph.osd.3.smithi046.stderr:2023-12-16T21:13:21.453+0000 7fefdce6f640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:21.468 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-12-16T21:13:21.468+0000 7f0bdc93e640 -1 osd.0 0 waiting for initial osdmap 2023-12-16T21:13:21.478 INFO:tasks.ceph.osd.0.smithi046.stderr:2023-12-16T21:13:21.478+0000 7f0bd4f77640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:21.503 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-12-16T21:13:21.503+0000 7f318e5b9640 -1 osd.1 0 waiting for initial osdmap 2023-12-16T21:13:21.513 INFO:tasks.ceph.osd.1.smithi046.stderr:2023-12-16T21:13:21.513+0000 7f3186bf2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:21.521 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-12-16T21:13:21.521+0000 7f5dc5e09640 -1 osd.2 0 waiting for initial osdmap 2023-12-16T21:13:21.534 INFO:tasks.ceph.osd.2.smithi046.stderr:2023-12-16T21:13:21.534+0000 7f5dbe442640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:22.031 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:22.062 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-12-16T21:13:22.062+0000 7fb071527640 -1 osd.6 0 waiting for initial osdmap 2023-12-16T21:13:22.075 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-12-16T21:13:22.074+0000 7ff01762b640 -1 osd.4 0 waiting for initial osdmap 2023-12-16T21:13:22.080 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-12-16T21:13:22.080+0000 7f2b26410640 -1 osd.7 0 waiting for initial osdmap 2023-12-16T21:13:22.081 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-12-16T21:13:22.081+0000 7fb069b60640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:22.085 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-12-16T21:13:22.085+0000 7ff00fc64640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:22.096 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-12-16T21:13:22.096+0000 7f2b1ea49640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:22.101 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-12-16T21:13:22.101+0000 7f4f54c3d640 -1 osd.5 0 waiting for initial osdmap 2023-12-16T21:13:22.113 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-12-16T21:13:22.113+0000 7f4f4d276640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:13:22.132 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.132+0000 7fcd4741d640 1 Processor -- start 2023-12-16T21:13:22.133 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.132+0000 7fcd4741d640 1 -- start start 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd4741d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcd4013ce90 0x7fcd400b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd4741d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013fef0 0x7fcd4015a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd4741d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400b4630 con 0x7fcd4013d270 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd4741d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd400b3d60 con 0x7fcd4013ce90 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd4741d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400b28c0 con 0x7fcd4013fef0 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd449b8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd4013d270 0x7fcd4013f7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47796/0 (socket says 172.21.15.46:47796) 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd449b8640 1 -- 172.21.15.46:0/1466449932 learned_addr learned my addr 172.21.15.46:0/1466449932 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:22.142 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd451b9640 1 --2- 172.21.15.46:0/1466449932 >> v2:172.21.15.121:3300/0 conn(0x7fcd4013ce90 0x7fcd400b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd459ba640 1 --2- 172.21.15.46:0/1466449932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013fef0 0x7fcd4015a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd37fff640 1 -- 172.21.15.46:0/1466449932 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911556537 0 0) 0x7fcd400b4630 con 0x7fcd4013d270 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd37fff640 1 -- 172.21.15.46:0/1466449932 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd2c003520 con 0x7fcd4013d270 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd37fff640 1 -- 172.21.15.46:0/1466449932 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3268026874 0 0) 0x7fcd2c003520 con 0x7fcd4013d270 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd459ba640 1 -- 172.21.15.46:0/1466449932 >> v2:172.21.15.121:3300/0 conn(0x7fcd4013ce90 msgr2=0x7fcd400b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd459ba640 1 --2- 172.21.15.46:0/1466449932 >> v2:172.21.15.121:3300/0 conn(0x7fcd4013ce90 0x7fcd400b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd459ba640 1 -- 172.21.15.46:0/1466449932 >> v1:172.21.15.46:6789/0 conn(0x7fcd4013d270 legacy=0x7fcd4013f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd459ba640 1 -- 172.21.15.46:0/1466449932 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4015b3e0 con 0x7fcd4013fef0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.133+0000 7fcd451b9640 1 --2- 172.21.15.46:0/1466449932 >> v2:172.21.15.121:3300/0 conn(0x7fcd4013ce90 0x7fcd400b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.134+0000 7fcd459ba640 1 --2- 172.21.15.46:0/1466449932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013fef0 0x7fcd4015a620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009d90 tx=0x7fcd3802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e6435439039927c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.134+0000 7fcd37fff640 1 -- 172.21.15.46:0/1466449932 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30001670 con 0x7fcd4013fef0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.134+0000 7fcd37fff640 1 -- 172.21.15.46:0/1466449932 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3802c8a0 con 0x7fcd4013fef0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.134+0000 7fcd37fff640 1 -- 172.21.15.46:0/1466449932 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3802cc30 con 0x7fcd4013fef0 2023-12-16T21:13:22.143 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 -- 172.21.15.46:0/1466449932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013fef0 msgr2=0x7fcd4015a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 --2- 172.21.15.46:0/1466449932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013fef0 0x7fcd4015a620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009d90 tx=0x7fcd3802bd70 comp rx=0 tx=0).stop 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 -- 172.21.15.46:0/1466449932 shutdown_connections 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 --2- 172.21.15.46:0/1466449932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013fef0 0x7fcd4015a620 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 --2- 172.21.15.46:0/1466449932 >> v2:172.21.15.121:3300/0 conn(0x7fcd4013ce90 0x7fcd400b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 -- 172.21.15.46:0/1466449932 >> 172.21.15.46:0/1466449932 conn(0x7fcd400b0c60 msgr2=0x7fcd400b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 -- 172.21.15.46:0/1466449932 shutdown_connections 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.135+0000 7fcd4741d640 1 -- 172.21.15.46:0/1466449932 wait complete. 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.136+0000 7fcd4741d640 1 Processor -- start 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.136+0000 7fcd4741d640 1 -- start start 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd4741d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4013ce90 0x7fcd401edff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd4741d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013d270 0x7fcd40158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd4741d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 0x7fcd40071900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd4741d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40153c20 con 0x7fcd4013fef0 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd4741d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd401f17d0 con 0x7fcd4013ce90 2023-12-16T21:13:22.144 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd4741d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd401f1ab0 con 0x7fcd4013d270 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd459ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 0x7fcd40071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd459ba640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 0x7fcd40071900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48144/0 (socket says 172.21.15.46:48144) 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd459ba640 1 -- 172.21.15.46:0/86084084 learned_addr learned my addr 172.21.15.46:0/86084084 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd451b9640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4013ce90 0x7fcd401edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd449b8640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013d270 0x7fcd40158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.137+0000 7fcd451b9640 1 -- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013d270 msgr2=0x7fcd40158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd451b9640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013d270 0x7fcd40158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd451b9640 1 -- 172.21.15.46:0/86084084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 msgr2=0x7fcd40071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd451b9640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 0x7fcd40071900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd451b9640 1 -- 172.21.15.46:0/86084084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c009400 con 0x7fcd4013ce90 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd459ba640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 0x7fcd40071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd451b9640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4013ce90 0x7fcd401edff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00e900 tx=0x7fcd3c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00ca00 con 0x7fcd4013ce90 2023-12-16T21:13:22.145 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd3c00cb60 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd38009a70 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c013470 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.138+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd40154e90 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.139+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd40069320 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.140+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 89177+0+0 (secure 0 0 0) 0x7fcd3c0135d0 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.140+0000 7fcd35ffb640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcd2406adb0 0x7fcd2406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.140+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fcd3c088820 con 0x7fcd4013ce90 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.140+0000 7fcd449b8640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcd2406adb0 0x7fcd2406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.141+0000 7fcd449b8640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcd2406adb0 0x7fcd2406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd40159e80 tx=0x7fcd30001dc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:22.146 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.145+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcd3c054880 con 0x7fcd4013ce90 2023-12-16T21:13:22.321 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.321+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd4006d8f0 con 0x7fcd4013ce90 2023-12-16T21:13:22.322 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.322+0000 7fcd35ffb640 1 -- 172.21.15.46:0/86084084 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fcd3c017090 con 0x7fcd4013ce90 2023-12-16T21:13:22.323 INFO:teuthology.misc.health.smithi046.stdout: 2023-12-16T21:13:22.323 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":12,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:19.956280+0000","last_up_change":"0.000000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:22.325 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcd2406adb0 msgr2=0x7fcd2406d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:22.325 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcd2406adb0 0x7fcd2406d270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd40159e80 tx=0x7fcd30001dc0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.325 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4013ce90 msgr2=0x7fcd401edff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:22.325 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4013ce90 0x7fcd401edff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00e900 tx=0x7fcd3c00edc0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.325 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 shutdown_connections 2023-12-16T21:13:22.325 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcd2406adb0 0x7fcd2406d270 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.326 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd4013fef0 0x7fcd40071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.326 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd4013d270 0x7fcd40158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.326 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 --2- 172.21.15.46:0/86084084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4013ce90 0x7fcd401edff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:22.326 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 >> 172.21.15.46:0/86084084 conn(0x7fcd400b0c60 msgr2=0x7fcd400b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:22.326 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 shutdown_connections 2023-12-16T21:13:22.326 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:22.324+0000 7fcd4741d640 1 -- 172.21.15.46:0/86084084 wait complete. 2023-12-16T21:13:22.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-16T21:13:27.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:13:27.075 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:13:29.141 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:29.245 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.245+0000 7f520ea51640 1 Processor -- start 2023-12-16T21:13:29.246 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.245+0000 7f520ea51640 1 -- start start 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520ea51640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f520815d190 0x7f520815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520ea51640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52080d1b10 0x7f52080d2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520ea51640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52080d1680 con 0x7f52080d1f50 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520ea51640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52080d0d90 con 0x7f52080d1b10 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520ea51640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52080b4060 con 0x7f520815d190 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f52077fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f52080d1b10 0x7f52080d2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520cfee640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f52080d1f50 0x7f52080b32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47812/0 (socket says 172.21.15.46:47812) 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f520cfee640 1 -- 172.21.15.46:0/33673530 learned_addr learned my addr 172.21.15.46:0/33673530 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f5207fff640 1 --2- 172.21.15.46:0/33673530 >> v2:172.21.15.121:3300/0 conn(0x7f520815d190 0x7f520815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f52077fe640 1 -- 172.21.15.46:0/33673530 >> v2:172.21.15.121:3300/0 conn(0x7f520815d190 msgr2=0x7f520815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f52077fe640 1 --2- 172.21.15.46:0/33673530 >> v2:172.21.15.121:3300/0 conn(0x7f520815d190 0x7f520815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.247 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f52077fe640 1 -- 172.21.15.46:0/33673530 >> v1:172.21.15.46:6789/0 conn(0x7f52080d1f50 legacy=0x7f52080b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.248 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.246+0000 7f52077fe640 1 -- 172.21.15.46:0/33673530 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52080d1030 con 0x7f52080d1b10 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.247+0000 7f52077fe640 1 --2- 172.21.15.46:0/33673530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1b10 0x7f52080d2f10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f52080d0d10 tx=0x7f51fc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b73e02b9b6ecdc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.247+0000 7f5206ffd640 1 -- 172.21.15.46:0/33673530 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc030610 con 0x7f52080d1b10 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.247+0000 7f5206ffd640 1 -- 172.21.15.46:0/33673530 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc030770 con 0x7f52080d1b10 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.247+0000 7f5206ffd640 1 -- 172.21.15.46:0/33673530 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc030b00 con 0x7f52080d1b10 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 -- 172.21.15.46:0/33673530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1b10 msgr2=0x7f52080d2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 --2- 172.21.15.46:0/33673530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1b10 0x7f52080d2f10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f52080d0d10 tx=0x7f51fc02b960 comp rx=0 tx=0).stop 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 -- 172.21.15.46:0/33673530 shutdown_connections 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 --2- 172.21.15.46:0/33673530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1b10 0x7f52080d2f10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.249 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 --2- 172.21.15.46:0/33673530 >> v2:172.21.15.121:3300/0 conn(0x7f520815d190 0x7f520815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.250 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 -- 172.21.15.46:0/33673530 >> 172.21.15.46:0/33673530 conn(0x7f520808cbe0 msgr2=0x7f520809c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:29.250 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 -- 172.21.15.46:0/33673530 shutdown_connections 2023-12-16T21:13:29.250 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.248+0000 7f520ea51640 1 -- 172.21.15.46:0/33673530 wait complete. 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.249+0000 7f520ea51640 1 Processor -- start 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.249+0000 7f520ea51640 1 -- start start 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520ea51640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52080d1b10 0x7f52080b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520ea51640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 0x7f52080b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520ea51640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f520815d190 0x7f520814f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520ea51640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208159220 con 0x7f52080d1f50 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520ea51640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208159500 con 0x7f52080d1b10 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520ea51640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52081597e0 con 0x7f520815d190 2023-12-16T21:13:29.252 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 0x7f52080b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 0x7f52080b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48164/0 (socket says 172.21.15.46:48164) 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 -- 172.21.15.46:0/1518415916 learned_addr learned my addr 172.21.15.46:0/1518415916 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f5207fff640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52080d1b10 0x7f52080b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f520cfee640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f520815d190 0x7f520814f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 -- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f520815d190 msgr2=0x7f520814f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f520815d190 0x7f520814f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 -- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52080d1b10 msgr2=0x7f52080b80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52080d1b10 0x7f52080b80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.250+0000 7f52077fe640 1 -- 172.21.15.46:0/1518415916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc009400 con 0x7f52080d1f50 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.251+0000 7f52077fe640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 0x7f52080b0ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009890 tx=0x7f51fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.251+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc03e040 con 0x7f52080d1f50 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.251+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51fc039cb0 con 0x7f52080d1f50 2023-12-16T21:13:29.253 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.251+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5208159ac0 con 0x7f52080d1f50 2023-12-16T21:13:29.254 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.251+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51fc044630 con 0x7f52080d1f50 2023-12-16T21:13:29.254 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.251+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52080b87c0 con 0x7f52080d1f50 2023-12-16T21:13:29.255 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.252+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5208069320 con 0x7f52080d1f50 2023-12-16T21:13:29.255 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.253+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f51fc044790 con 0x7f52080d1f50 2023-12-16T21:13:29.255 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.253+0000 7f5204ff9640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f51f406ae40 0x7f51f406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.255 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.253+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f51fc0ab470 con 0x7f52080d1f50 2023-12-16T21:13:29.255 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.253+0000 7f5207fff640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f51f406ae40 0x7f51f406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.255 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.254+0000 7f5207fff640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f51f406ae40 0x7f51f406d300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5208073930 tx=0x7f51f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.258 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.258+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f51fc0769b0 con 0x7f52080d1f50 2023-12-16T21:13:29.437 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.436+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52080b87c0 con 0x7f52080d1f50 2023-12-16T21:13:29.438 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.437+0000 7f5204ff9640 1 -- 172.21.15.46:0/1518415916 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f52080b87c0 con 0x7f52080d1f50 2023-12-16T21:13:29.438 INFO:teuthology.misc.health.smithi046.stdout: 2023-12-16T21:13:29.438 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":17,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:26.979885+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:29.440 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.439+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f51f406ae40 msgr2=0x7f51f406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.440 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.439+0000 7f520ea51640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f51f406ae40 0x7f51f406d300 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5208073930 tx=0x7f51f8008040 comp rx=0 tx=0).stop 2023-12-16T21:13:29.440 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.439+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 msgr2=0x7f52080b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.439+0000 7f520ea51640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 0x7f52080b0ed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009890 tx=0x7f51fc004730 comp rx=0 tx=0).stop 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 shutdown_connections 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f51f406ae40 0x7f51f406d300 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52080d1f50 0x7f52080b0ed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f520815d190 0x7f520814f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 --2- 172.21.15.46:0/1518415916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52080d1b10 0x7f52080b80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 >> 172.21.15.46:0/1518415916 conn(0x7f520808cbe0 msgr2=0x7f520815db10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 shutdown_connections 2023-12-16T21:13:29.441 INFO:teuthology.misc.health.smithi046.stderr:2023-12-16T21:13:29.440+0000 7f520ea51640 1 -- 172.21.15.46:0/1518415916 wait complete. 2023-12-16T21:13:29.450 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-16T21:13:29.450 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-16T21:13:29.450 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-16T21:13:29.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-16T21:13:29.453 INFO:tasks.ceph.mds.a:Started 2023-12-16T21:13:29.453 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-16T21:13:29.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-16T21:13:29.455 INFO:tasks.ceph.mds.c:Started 2023-12-16T21:13:29.456 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-16T21:13:29.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-16T21:13:29.459 INFO:tasks.ceph.mds.b:Started 2023-12-16T21:13:29.459 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-16T21:13:29.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-16T21:13:29.461 INFO:tasks.ceph.mds.d:Started 2023-12-16T21:13:29.461 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-16T21:13:29.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:29.535 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-12-16T21:13:29.535 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-12-16T21:13:29.543 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-12-16T21:13:29.544 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2023-12-16T21:13:29.544 INFO:tasks.ceph.mds.b.smithi121.stderr:2023-12-16T21:13:29.544+0000 7f4fed362640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6835/0: (98) Address already in use 2023-12-16T21:13:29.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.572+0000 7f5d28615640 1 Processor -- start 2023-12-16T21:13:29.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.572+0000 7f5d28615640 1 -- start start 2023-12-16T21:13:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d28615640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d2015d330 0x7f5d200b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d28615640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d200d2910 0x7f5d200d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d28615640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d200b2b10 con 0x7f5d200d3730 2023-12-16T21:13:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d28615640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d200d3b50 con 0x7f5d2015d330 2023-12-16T21:13:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d28615640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200d1590 con 0x7f5d200d2910 2023-12-16T21:13:29.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d26bb2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5d200d3730 0x7f5d200b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47836/0 (socket says 172.21.15.46:47836) 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d26bb2640 1 -- 172.21.15.46:0/247807820 learned_addr learned my addr 172.21.15.46:0/247807820 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d263b1640 1 --2- 172.21.15.46:0/247807820 >> v2:172.21.15.121:3300/0 conn(0x7f5d2015d330 0x7f5d200b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d25bb0640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d200d2910 0x7f5d200d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730897844 0 0) 0x7f5d200b2b10 con 0x7f5d200d3730 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d0c003520 con 0x7f5d200d3730 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1419181816 0 0) 0x7f5d0c003520 con 0x7f5d200d3730 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d200d2910 msgr2=0x7f5d200d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d200d2910 0x7f5d200d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 >> v2:172.21.15.121:3300/0 conn(0x7f5d2015d330 msgr2=0x7f5d200b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 --2- 172.21.15.46:0/247807820 >> v2:172.21.15.121:3300/0 conn(0x7f5d2015d330 0x7f5d200b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d200d1870 con 0x7f5d200d3730 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f5d180019f0 con 0x7f5d200d3730 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d263b1640 1 --2- 172.21.15.46:0/247807820 >> v2:172.21.15.121:3300/0 conn(0x7f5d2015d330 0x7f5d200b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 >> v1:172.21.15.46:6789/0 conn(0x7f5d200d3730 legacy=0x7f5d200b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c003970 0x7f5d0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d0c0047c0 0x7f5d0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c005180 0x7f5d0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200d3b50 con 0x7f5d0c003970 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.573+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d200d1590 con 0x7f5d0c005180 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c00bf10 con 0x7f5d0c0047c0 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d25bb0640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d200d2910 0x7f5d200d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d263b1640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c003970 0x7f5d0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d26bb2640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c005180 0x7f5d0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d25bb0640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d0c0047c0 0x7f5d0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d263b1640 1 -- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d0c0047c0 msgr2=0x7f5d0c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d263b1640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d0c0047c0 0x7f5d0c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d263b1640 1 -- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c005180 msgr2=0x7f5d0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d263b1640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c005180 0x7f5d0c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.574+0000 7f5d263b1640 1 -- 172.21.15.46:0/247807820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c003520 con 0x7f5d0c003970 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d263b1640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c003970 0x7f5d0c003d90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009cc0 tx=0x7f5d0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73da80dd4d1c579c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c02c960 con 0x7f5d0c003970 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c02cac0 con 0x7f5d0c003970 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d253af640 1 -- 172.21.15.46:0/247807820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c02ce70 con 0x7f5d0c003970 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d28615640 1 -- 172.21.15.46:0/247807820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c003970 msgr2=0x7f5d0c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d28615640 1 --2- 172.21.15.46:0/247807820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d0c003970 0x7f5d0c003d90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009cc0 tx=0x7f5d0c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d26bb2640 1 -- 172.21.15.46:0/247807820 reap_dead start 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d28615640 1 -- 172.21.15.46:0/247807820 shutdown_connections 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d28615640 1 -- 172.21.15.46:0/247807820 >> 172.21.15.46:0/247807820 conn(0x7f5d20072020 msgr2=0x7f5d20072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d28615640 1 -- 172.21.15.46:0/247807820 shutdown_connections 2023-12-16T21:13:29.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.575+0000 7f5d28615640 1 -- 172.21.15.46:0/247807820 wait complete. 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 Processor -- start 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 -- start start 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c003970 0x7f5d20156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d201f6e90 0x7f5d201f5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 0x7f5d200c8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201568c0 con 0x7f5d20153560 2023-12-16T21:13:29.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201552e0 con 0x7f5d0c003970 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d28615640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d201555c0 con 0x7f5d201f6e90 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d26bb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 0x7f5d200c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d26bb2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 0x7f5d200c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48216/0 (socket says 172.21.15.46:48216) 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.577+0000 7f5d26bb2640 1 -- 172.21.15.46:0/3020935763 learned_addr learned my addr 172.21.15.46:0/3020935763 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d25bb0640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d201f6e90 0x7f5d201f5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d263b1640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c003970 0x7f5d20156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d26bb2640 1 -- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d201f6e90 msgr2=0x7f5d201f5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d26bb2640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d201f6e90 0x7f5d201f5a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d26bb2640 1 -- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c003970 msgr2=0x7f5d20156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d26bb2640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c003970 0x7f5d20156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d26bb2640 1 -- 172.21.15.46:0/3020935763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d201558a0 con 0x7f5d20153560 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d26bb2640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 0x7f5d200c8680 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d2008c510 tx=0x7f5d18007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1800ab00 con 0x7f5d20153560 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1800aca0 con 0x7f5d20153560 2023-12-16T21:13:29.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d20155b80 con 0x7f5d20153560 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d18013630 con 0x7f5d20153560 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.578+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d201f5f40 con 0x7f5d20153560 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.579+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d201560a0 con 0x7f5d20153560 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.580+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5d18013790 con 0x7f5d20153560 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.580+0000 7f5d16ffd640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5d0406ae70 0x7f5d0406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.580+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5d18089980 con 0x7f5d20153560 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.580+0000 7f5d263b1640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5d0406ae70 0x7f5d0406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.581+0000 7f5d263b1640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5d0406ae70 0x7f5d0406d330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c036000 tx=0x7f5d1c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.585+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5d201560a0 con 0x7f5d20153560 2023-12-16T21:13:29.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.765+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d201560a0 con 0x7f5d20153560 2023-12-16T21:13:29.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.766+0000 7f5d16ffd640 1 -- 172.21.15.46:0/3020935763 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f5d201560a0 con 0x7f5d20153560 2023-12-16T21:13:29.767 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:29.767 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:26.979885+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5d0406ae70 msgr2=0x7f5d0406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5d0406ae70 0x7f5d0406d330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c036000 tx=0x7f5d1c030000 comp rx=0 tx=0).stop 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 msgr2=0x7f5d200c8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 0x7f5d200c8680 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d2008c510 tx=0x7f5d18007b00 comp rx=0 tx=0).stop 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 shutdown_connections 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5d0406ae70 0x7f5d0406d330 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5d20153560 0x7f5d200c8680 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d201f6e90 0x7f5d201f5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 --2- 172.21.15.46:0/3020935763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d0c003970 0x7f5d20156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 >> 172.21.15.46:0/3020935763 conn(0x7f5d20072020 msgr2=0x7f5d200d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:29.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 shutdown_connections 2023-12-16T21:13:29.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.768+0000 7f5d28615640 1 -- 172.21.15.46:0/3020935763 wait complete. 2023-12-16T21:13:29.779 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:13:29.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:13:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.888+0000 7f1e8b46b640 1 Processor -- start 2023-12-16T21:13:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.888+0000 7f1e8b46b640 1 -- start start 2023-12-16T21:13:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.889+0000 7f1e8b46b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1e84157b00 0x7f1e840714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.889+0000 7f1e8b46b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8415b340 con 0x7f1e84071a00 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.889+0000 7f1e8b46b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e84071e60 con 0x7f1e8415af30 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.889+0000 7f1e8b46b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8406f510 con 0x7f1e84157b00 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.889+0000 7f1e88a06640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1e84071a00 0x7f1e8415a820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47846/0 (socket says 172.21.15.46:47846) 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.889+0000 7f1e88a06640 1 -- 172.21.15.46:0/2637206272 learned_addr learned my addr 172.21.15.46:0/2637206272 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e89207640 1 --2- 172.21.15.46:0/2637206272 >> v2:172.21.15.121:3300/0 conn(0x7f1e84157b00 0x7f1e840714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e83fff640 1 -- 172.21.15.46:0/2637206272 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951503888 0 0) 0x7f1e8415b340 con 0x7f1e84071a00 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e83fff640 1 -- 172.21.15.46:0/2637206272 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e6c003520 con 0x7f1e84071a00 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e89207640 1 -- 172.21.15.46:0/2637206272 >> v1:172.21.15.121:6789/0 conn(0x7f1e8415af30 legacy=0x7f1e840b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e83fff640 1 -- 172.21.15.46:0/2637206272 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580477034 0 0) 0x7f1e6c003520 con 0x7f1e84071a00 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e89207640 1 -- 172.21.15.46:0/2637206272 >> v1:172.21.15.46:6789/0 conn(0x7f1e84071a00 legacy=0x7f1e8415a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.890+0000 7f1e89207640 1 -- 172.21.15.46:0/2637206272 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e84157ee0 con 0x7f1e84157b00 2023-12-16T21:13:29.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e89207640 1 --2- 172.21.15.46:0/2637206272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e84157b00 0x7f1e840714c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0097a0 tx=0x7f1e7c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecd5b062cf0b6ab0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e83fff640 1 -- 172.21.15.46:0/2637206272 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e78001670 con 0x7f1e84157b00 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e83fff640 1 -- 172.21.15.46:0/2637206272 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7c02e050 con 0x7f1e84157b00 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e83fff640 1 -- 172.21.15.46:0/2637206272 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7c030710 con 0x7f1e84157b00 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e8b46b640 1 -- 172.21.15.46:0/2637206272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e84157b00 msgr2=0x7f1e840714c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/2637206272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e84157b00 0x7f1e840714c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c0097a0 tx=0x7f1e7c02b790 comp rx=0 tx=0).stop 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e8b46b640 1 -- 172.21.15.46:0/2637206272 shutdown_connections 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/2637206272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e84157b00 0x7f1e840714c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.891+0000 7f1e8b46b640 1 -- 172.21.15.46:0/2637206272 >> 172.21.15.46:0/2637206272 conn(0x7f1e840b1de0 msgr2=0x7f1e840b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.892+0000 7f1e8b46b640 1 -- 172.21.15.46:0/2637206272 shutdown_connections 2023-12-16T21:13:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.892+0000 7f1e8b46b640 1 -- 172.21.15.46:0/2637206272 wait complete. 2023-12-16T21:13:29.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 Processor -- start 2023-12-16T21:13:29.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 -- start start 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e84071a00 0x7f1e840af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 0x7f1e841478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e8415af30 0x7f1e840a8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841487e0 con 0x7f1e84157b00 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84152bd0 con 0x7f1e8415af30 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e8b46b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84147f40 con 0x7f1e84071a00 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.893+0000 7f1e88a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 0x7f1e841478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 0x7f1e841478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48220/0 (socket says 172.21.15.46:48220) 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 -- 172.21.15.46:0/1093544536 learned_addr learned my addr 172.21.15.46:0/1093544536 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e89207640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e84071a00 0x7f1e840af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e89a08640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e8415af30 0x7f1e840a8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 -- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e84071a00 msgr2=0x7f1e840af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e84071a00 0x7f1e840af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 -- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e8415af30 msgr2=0x7f1e840a8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e8415af30 0x7f1e840a8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 -- 172.21.15.46:0/1093544536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c009400 con 0x7f1e84157b00 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e88a06640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 0x7f1e841478b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e780011e0 tx=0x7f1e7800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e78005ad0 con 0x7f1e84157b00 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e78005c50 con 0x7f1e84157b00 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e84148220 con 0x7f1e84157b00 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e780072b0 con 0x7f1e84157b00 2023-12-16T21:13:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.894+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e84145950 con 0x7f1e84157b00 2023-12-16T21:13:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.895+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84145b70 con 0x7f1e84157b00 2023-12-16T21:13:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.896+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1e78007410 con 0x7f1e84157b00 2023-12-16T21:13:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.896+0000 7f1e81ffb640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1e7006ae40 0x7f1e7006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.896+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1e78088940 con 0x7f1e84157b00 2023-12-16T21:13:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.896+0000 7f1e89207640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1e7006ae40 0x7f1e7006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.897+0000 7f1e89207640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1e7006ae40 0x7f1e7006d300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c034000 tx=0x7f1e7c02f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:29.900+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1e78053df0 con 0x7f1e84157b00 2023-12-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.075+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e84067420 con 0x7f1e84157b00 2023-12-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.075+0000 7f1e81ffb640 1 -- 172.21.15.46:0/1093544536 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e78017020 con 0x7f1e84157b00 2023-12-16T21:13:30.076 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1e7006ae40 msgr2=0x7f1e7006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1e7006ae40 0x7f1e7006d300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c034000 tx=0x7f1e7c02f000 comp rx=0 tx=0).stop 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 msgr2=0x7f1e841478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 0x7f1e841478b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e780011e0 tx=0x7f1e7800cbe0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 shutdown_connections 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1e7006ae40 0x7f1e7006d300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e84157b00 0x7f1e841478b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1e8415af30 0x7f1e840a8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 --2- 172.21.15.46:0/1093544536 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1e84071a00 0x7f1e840af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 >> 172.21.15.46:0/1093544536 conn(0x7f1e840b1de0 msgr2=0x7f1e840afd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 shutdown_connections 2023-12-16T21:13:30.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.077+0000 7f1e8b46b640 1 -- 172.21.15.46:0/1093544536 wait complete. 2023-12-16T21:13:30.088 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:13:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.194+0000 7f71115c2640 1 Processor -- start 2023-12-16T21:13:30.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.194+0000 7f71115c2640 1 -- start start 2023-12-16T21:13:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.195+0000 7f71115c2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 0x7f710c0d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.195+0000 7f71115c2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0b3f10 con 0x7f710c15b130 2023-12-16T21:13:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.195+0000 7f71115c2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710c0d0a50 con 0x7f710c15f760 2023-12-16T21:13:30.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.195+0000 7f71115c2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c15fd20 con 0x7f710c0cf320 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.195+0000 7f710affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f710c15b130 0x7f710c0cec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:47856/0 (socket says 172.21.15.46:47856) 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.195+0000 7f710affd640 1 -- 172.21.15.46:0/2360590209 learned_addr learned my addr 172.21.15.46:0/2360590209 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 0x7f710c0d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770567642 0 0) 0x7f710c0b3f10 con 0x7f710c15b130 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f8003520 con 0x7f710c15b130 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945052275 0 0) 0x7f70f8003520 con 0x7f710c15b130 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 msgr2=0x7f710c0d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 0x7f710c0d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 >> v1:172.21.15.121:6789/0 conn(0x7f710c15f760 legacy=0x7f710c0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c0d0d30 con 0x7f710c15b130 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f7104001670 con 0x7f710c15b130 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 >> v1:172.21.15.46:6789/0 conn(0x7f710c15b130 legacy=0x7f710c0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80039a0 0x7f70f8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f80047c0 0x7f70f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 0x7f70f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c15fd20 con 0x7f70f80047c0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f800ba90 con 0x7f70f80039a0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f800bc50 con 0x7f710c0cf320 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f80047c0 0x7f70f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710affd640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80039a0 0x7f70f8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710b7fe640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 0x7f70f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 -- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 msgr2=0x7f70f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c0cf320 0x7f70f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 -- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80039a0 msgr2=0x7f70f8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80039a0 0x7f70f8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.196+0000 7f710a7fc640 1 -- 172.21.15.46:0/2360590209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f8003520 con 0x7f70f80047c0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.197+0000 7f710a7fc640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f80047c0 0x7f70f8004c40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009a90 tx=0x7f70f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80bd38640acc8f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.197+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f402c680 con 0x7f70f80047c0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.197+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f402c7e0 con 0x7f70f80047c0 2023-12-16T21:13:30.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.197+0000 7f7109ffb640 1 -- 172.21.15.46:0/2360590209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f402cb70 con 0x7f70f80047c0 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f71115c2640 1 -- 172.21.15.46:0/2360590209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f80047c0 msgr2=0x7f70f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f71115c2640 1 --2- 172.21.15.46:0/2360590209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70f80047c0 0x7f70f8004c40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009a90 tx=0x7f70f80057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f710b7fe640 1 -- 172.21.15.46:0/2360590209 reap_dead start 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f71115c2640 1 -- 172.21.15.46:0/2360590209 shutdown_connections 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f71115c2640 1 -- 172.21.15.46:0/2360590209 >> 172.21.15.46:0/2360590209 conn(0x7f710c0b2c10 msgr2=0x7f710c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f71115c2640 1 -- 172.21.15.46:0/2360590209 shutdown_connections 2023-12-16T21:13:30.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.198+0000 7f71115c2640 1 -- 172.21.15.46:0/2360590209 wait complete. 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.199+0000 7f71115c2640 1 Processor -- start 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.199+0000 7f71115c2640 1 -- start start 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71115c2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80047c0 0x7f710c0c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71115c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 0x7f710c15ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71115c2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c15d6b0 0x7f710c15e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71115c2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c14fe00 con 0x7f710c15b810 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71115c2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c0c8c30 con 0x7f70f80047c0 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71115c2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c06d250 con 0x7f710c15d6b0 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 0x7f710c15ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 0x7f710c15ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48226/0 (socket says 172.21.15.46:48226) 2023-12-16T21:13:30.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 -- 172.21.15.46:0/1998609736 learned_addr learned my addr 172.21.15.46:0/1998609736 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710affd640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80047c0 0x7f710c0c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710b7fe640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c15d6b0 0x7f710c15e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 -- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c15d6b0 msgr2=0x7f710c15e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c15d6b0 0x7f710c15e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 -- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80047c0 msgr2=0x7f710c0c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80047c0 0x7f710c0c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 -- 172.21.15.46:0/1998609736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c06d530 con 0x7f710c15b810 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f710a7fc640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 0x7f710c15ddf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70f402c220 tx=0x7f70f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4003810 con 0x7f710c15b810 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.200+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f4003970 con 0x7f710c15b810 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.201+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c0c5a90 con 0x7f710c15b810 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.201+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4003ce0 con 0x7f710c15b810 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.201+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c0c6050 con 0x7f710c15b810 2023-12-16T21:13:30.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.201+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c0c6210 con 0x7f710c15b810 2023-12-16T21:13:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.202+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f70f4034440 con 0x7f710c15b810 2023-12-16T21:13:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.202+0000 7f71037fe640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70ec06ae40 0x7f70ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.202+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f70f40ab130 con 0x7f710c15b810 2023-12-16T21:13:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.202+0000 7f710affd640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70ec06ae40 0x7f70ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:30.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.203+0000 7f710affd640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70ec06ae40 0x7f70ec06d300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7104001eb0 tx=0x7f7104009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:30.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.207+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f70f4076670 con 0x7f710c15b810 2023-12-16T21:13:30.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.418+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f710c067fc0 con 0x7f710c15b810 2023-12-16T21:13:30.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.418+0000 7f71037fe640 1 -- 172.21.15.46:0/1998609736 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7f70f40044e0 con 0x7f710c15b810 2023-12-16T21:13:30.419 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:30.419 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-12-16T21:13:30.419 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70ec06ae40 msgr2=0x7f70ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70ec06ae40 0x7f70ec06d300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7104001eb0 tx=0x7f7104009040 comp rx=0 tx=0).stop 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 msgr2=0x7f710c15ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 0x7f710c15ddf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70f402c220 tx=0x7f70f4004730 comp rx=0 tx=0).stop 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 shutdown_connections 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70ec06ae40 0x7f70ec06d300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f710c15b810 0x7f710c15ddf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f710c15d6b0 0x7f710c15e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 --2- 172.21.15.46:0/1998609736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f80047c0 0x7f710c0c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 >> 172.21.15.46:0/1998609736 conn(0x7f710c0b2c10 msgr2=0x7f710c0b5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 shutdown_connections 2023-12-16T21:13:30.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:30.420+0000 7f71115c2640 1 -- 172.21.15.46:0/1998609736 wait complete. 2023-12-16T21:13:32.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:13:32.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.550+0000 7f06be6e7640 1 Processor -- start 2023-12-16T21:13:32.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.550+0000 7f06be6e7640 1 -- start start 2023-12-16T21:13:32.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06be6e7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 0x7f06b815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06be6e7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06b80ceb70 0x7f06b80cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06be6e7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b8071770 con 0x7f06b80b4370 2023-12-16T21:13:32.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06be6e7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06b80ccdc0 con 0x7f06b80ceb70 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06be6e7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80cd0a0 con 0x7f06b8144980 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06b80ceb70 0x7f06b80cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 0x7f06b815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06b80ceb70 0x7f06b80cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:51402/0 (socket says 172.21.15.46:51402) 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b7fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 0x7f06b815c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:47762/0 (socket says 172.21.15.46:47762) 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b77fe640 1 -- 172.21.15.46:0/2316663366 learned_addr learned my addr 172.21.15.46:0/2316663366 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b77fe640 1 -- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 msgr2=0x7f06b815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b77fe640 1 --2- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 0x7f06b815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.551+0000 7f06b77fe640 1 -- 172.21.15.46:0/2316663366 >> v1:172.21.15.121:6789/0 conn(0x7f06b80b4370 legacy=0x7f06b8071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.552+0000 7f06b77fe640 1 -- 172.21.15.46:0/2316663366 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80cd380 con 0x7f06b80ceb70 2023-12-16T21:13:32.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.552+0000 7f06b6ffd640 1 -- 172.21.15.46:0/2316663366 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278013480 0 0) 0x7f06b8071770 con 0x7f06b80b4370 2023-12-16T21:13:32.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.552+0000 7f06b7fff640 1 --2- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 0x7f06b815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:32.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.552+0000 7f06b77fe640 1 --2- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b80ceb70 0x7f06b80cc450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009870 tx=0x7f06a802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aa3ca0283b80933 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:32.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06b6ffd640 1 -- 172.21.15.46:0/2316663366 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8030840 con 0x7f06b80ceb70 2023-12-16T21:13:32.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06b6ffd640 1 -- 172.21.15.46:0/2316663366 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a80309a0 con 0x7f06b80ceb70 2023-12-16T21:13:32.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06b6ffd640 1 -- 172.21.15.46:0/2316663366 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8030d30 con 0x7f06b80ceb70 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 -- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b80ceb70 msgr2=0x7f06b80cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 --2- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b80ceb70 0x7f06b80cc450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f06a8009870 tx=0x7f06a802b980 comp rx=0 tx=0).stop 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 -- 172.21.15.46:0/2316663366 shutdown_connections 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 --2- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b80ceb70 0x7f06b80cc450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 --2- 172.21.15.46:0/2316663366 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b8144980 0x7f06b815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 -- 172.21.15.46:0/2316663366 >> 172.21.15.46:0/2316663366 conn(0x7f06b806a4d0 msgr2=0x7f06b806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 -- 172.21.15.46:0/2316663366 shutdown_connections 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.553+0000 7f06be6e7640 1 -- 172.21.15.46:0/2316663366 wait complete. 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.554+0000 7f06be6e7640 1 Processor -- start 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 -- start start 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b80b4370 0x7f06b80afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 0x7f06b81580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b8144980 0x7f06b8158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8158b50 con 0x7f06b80ceb70 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80b0410 con 0x7f06b8144980 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06be6e7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b80b06f0 con 0x7f06b80b4370 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06b77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 0x7f06b81580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06b77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 0x7f06b81580d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54244/0 (socket says 172.21.15.46:54244) 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.555+0000 7f06b77fe640 1 -- 172.21.15.46:0/3805774049 learned_addr learned my addr 172.21.15.46:0/3805774049 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b7fff640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b80b4370 0x7f06b80afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06bcc84640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b8144980 0x7f06b8158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b77fe640 1 -- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b80b4370 msgr2=0x7f06b80afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b77fe640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b80b4370 0x7f06b80afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b77fe640 1 -- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b8144980 msgr2=0x7f06b8158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b77fe640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b8144980 0x7f06b8158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b77fe640 1 -- 172.21.15.46:0/3805774049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac009400 con 0x7f06b80ceb70 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b77fe640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 0x7f06b81580d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f06a8000c00 tx=0x7f06a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:32.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8039c90 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a8039df0 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a80094d0 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a8043640 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.556+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b80b0cb0 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.557+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b8069400 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.558+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f06a80437a0 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.558+0000 7f06b4ff9640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f06a406ae40 0x7f06a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.558+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f06a803e070 con 0x7f06b80ceb70 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.558+0000 7f06b7fff640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f06a406ae40 0x7f06a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.559+0000 7f06b7fff640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f06a406ae40 0x7f06a406d300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009c30 tx=0x7f06ac009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:32.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.563+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f06a8077660 con 0x7f06b80ceb70 2023-12-16T21:13:32.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.776+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06b80b4b20 con 0x7f06b80ceb70 2023-12-16T21:13:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.776+0000 7f06b4ff9640 1 -- 172.21.15.46:0/3805774049 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7f06a807ac80 con 0x7f06b80ceb70 2023-12-16T21:13:32.777 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:32.777 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-12-16T21:13:32.777 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f06a406ae40 msgr2=0x7f06a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f06a406ae40 0x7f06a406d300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009c30 tx=0x7f06ac009190 comp rx=0 tx=0).stop 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 msgr2=0x7f06b81580d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 0x7f06b81580d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f06a8000c00 tx=0x7f06a8004730 comp rx=0 tx=0).stop 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 shutdown_connections 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f06a406ae40 0x7f06a406d300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06b80ceb70 0x7f06b81580d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06b8144980 0x7f06b8158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 --2- 172.21.15.46:0/3805774049 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06b80b4370 0x7f06b80afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 >> 172.21.15.46:0/3805774049 conn(0x7f06b806a4d0 msgr2=0x7f06b815df10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 shutdown_connections 2023-12-16T21:13:32.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.778+0000 7f06be6e7640 1 -- 172.21.15.46:0/3805774049 wait complete. 2023-12-16T21:13:32.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:32.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 Processor -- start 2023-12-16T21:13:32.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 -- start start 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fedb415d2d0 0x7fedb40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb40ce650 0x7fedb40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb41614e0 con 0x7fedb4160f70 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb41617c0 con 0x7fedb415d2d0 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.896+0000 7fedb991d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb4161aa0 con 0x7fedb40ce650 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedabfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fedb4160f70 0x7fedb40cdf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33512/0 (socket says 172.21.15.46:33512) 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedabfff640 1 -- 172.21.15.46:0/4135431661 learned_addr learned my addr 172.21.15.46:0/4135431661 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb2ffd640 1 --2- 172.21.15.46:0/4135431661 >> v2:172.21.15.121:3300/0 conn(0x7fedb415d2d0 0x7fedb40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb37fe640 1 --2- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb40ce650 0x7fedb40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb27fc640 1 -- 172.21.15.46:0/4135431661 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294742821 0 0) 0x7fedb41614e0 con 0x7fedb4160f70 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb27fc640 1 -- 172.21.15.46:0/4135431661 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0003520 con 0x7fedb4160f70 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb27fc640 1 -- 172.21.15.46:0/4135431661 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 988809133 0 0) 0x7feda0003520 con 0x7fedb4160f70 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb2ffd640 1 -- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb40ce650 msgr2=0x7fedb40d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:32.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb2ffd640 1 --2- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb40ce650 0x7fedb40d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb2ffd640 1 -- 172.21.15.46:0/4135431661 >> v1:172.21.15.46:6789/0 conn(0x7fedb4160f70 legacy=0x7fedb40cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.897+0000 7fedb2ffd640 1 -- 172.21.15.46:0/4135431661 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb4161d80 con 0x7fedb415d2d0 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.898+0000 7fedb2ffd640 1 --2- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb415d2d0 0x7fedb40b8b90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feda40097a0 tx=0x7feda402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f73b4b58ced2997 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.898+0000 7fedb27fc640 1 -- 172.21.15.46:0/4135431661 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed9c001670 con 0x7fedb415d2d0 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.898+0000 7fedb27fc640 1 -- 172.21.15.46:0/4135431661 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda402e050 con 0x7fedb415d2d0 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.898+0000 7fedb27fc640 1 -- 172.21.15.46:0/4135431661 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda4030710 con 0x7fedb415d2d0 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 -- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb415d2d0 msgr2=0x7fedb40b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 --2- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb415d2d0 0x7fedb40b8b90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7feda40097a0 tx=0x7feda402b790 comp rx=0 tx=0).stop 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 -- 172.21.15.46:0/4135431661 shutdown_connections 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 --2- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb40ce650 0x7fedb40d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 --2- 172.21.15.46:0/4135431661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb415d2d0 0x7fedb40b8b90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 -- 172.21.15.46:0/4135431661 >> 172.21.15.46:0/4135431661 conn(0x7fedb4072230 msgr2=0x7fedb40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 -- 172.21.15.46:0/4135431661 shutdown_connections 2023-12-16T21:13:32.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.899+0000 7fedb991d640 1 -- 172.21.15.46:0/4135431661 wait complete. 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.900+0000 7fedb991d640 1 Processor -- start 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.900+0000 7fedb991d640 1 -- start start 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 0x7fedb40cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb415d2d0 0x7fedb40b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb4160f70 0x7fedb40b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40b2b60 con 0x7fedb40ce650 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40b2e40 con 0x7fedb4160f70 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40b3120 con 0x7fedb415d2d0 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 0x7fedb40cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 0x7fedb40cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54258/0 (socket says 172.21.15.46:54258) 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 -- 172.21.15.46:0/1246837280 learned_addr learned my addr 172.21.15.46:0/1246837280 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:32.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedabfff640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb415d2d0 0x7fedb40b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb37fe640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb4160f70 0x7fedb40b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 -- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb415d2d0 msgr2=0x7fedb40b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb415d2d0 0x7fedb40b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 -- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb4160f70 msgr2=0x7fedb40b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb4160f70 0x7fedb40b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 -- 172.21.15.46:0/1246837280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda4009400 con 0x7fedb40ce650 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb2ffd640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 0x7fedb40cb1f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feda4000c00 tx=0x7feda40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda403c620 con 0x7fedb40ce650 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feda403c7a0 con 0x7fedb40ce650 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb40b3400 con 0x7fedb40ce650 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda403cb50 con 0x7fedb40ce650 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.901+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb40ca420 con 0x7fedb40ce650 2023-12-16T21:13:32.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.902+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb4069400 con 0x7fedb40ce650 2023-12-16T21:13:32.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.903+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7feda4039070 con 0x7fedb40ce650 2023-12-16T21:13:32.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.903+0000 7fedb0ff9640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fed9406ae40 0x7fed9406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:32.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.903+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feda40aa990 con 0x7fedb40ce650 2023-12-16T21:13:32.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.903+0000 7fedabfff640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fed9406ae40 0x7fed9406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:32.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.904+0000 7fedabfff640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fed9406ae40 0x7fed9406d300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fedb41f63e0 tx=0x7fed9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:32.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:32.907+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7feda4075e20 con 0x7fedb40ce650 2023-12-16T21:13:33.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.084+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fedb40d2db0 con 0x7fedb40ce650 2023-12-16T21:13:33.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.085+0000 7fedb0ff9640 1 -- 172.21.15.46:0/1246837280 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12850 (secure 0 0 0) 0x7feda404a090 con 0x7fedb40ce650 2023-12-16T21:13:33.085 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:33.086 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:26.979885+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:33.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fed9406ae40 msgr2=0x7fed9406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fed9406ae40 0x7fed9406d300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fedb41f63e0 tx=0x7fed9c009040 comp rx=0 tx=0).stop 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 msgr2=0x7fedb40cb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 0x7fedb40cb1f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feda4000c00 tx=0x7feda40046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 shutdown_connections 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fed9406ae40 0x7fed9406d300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedb4160f70 0x7fedb40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fedb415d2d0 0x7fedb40b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 --2- 172.21.15.46:0/1246837280 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb40ce650 0x7fedb40cb1f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 >> 172.21.15.46:0/1246837280 conn(0x7fedb4072230 msgr2=0x7fedb40b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 shutdown_connections 2023-12-16T21:13:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.087+0000 7fedb991d640 1 -- 172.21.15.46:0/1246837280 wait complete. 2023-12-16T21:13:33.098 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:13:33.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:13:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.207+0000 7fd3ada72640 1 Processor -- start 2023-12-16T21:13:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.207+0000 7fd3ada72640 1 -- start start 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.207+0000 7fd3ada72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a80b1400 0x7fd3a815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3ada72640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a815e2a0 con 0x7fd3a815e730 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3ada72640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a815b240 con 0x7fd3a8143490 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3ada72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a815b520 con 0x7fd3a80b1400 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3a815e730 0x7fd3a815aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33520/0 (socket says 172.21.15.46:33520) 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a7fff640 1 -- 172.21.15.46:0/1163545891 learned_addr learned my addr 172.21.15.46:0/1163545891 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a6ffd640 1 --2- 172.21.15.46:0/1163545891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a80b1400 0x7fd3a815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758078485 0 0) 0x7fd3a815e2a0 con 0x7fd3a815e730 2023-12-16T21:13:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd390003520 con 0x7fd3a815e730 2023-12-16T21:13:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869472223 0 0) 0x7fd3a815b240 con 0x7fd3a8143490 2023-12-16T21:13:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3a815e2a0 con 0x7fd3a8143490 2023-12-16T21:13:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/1163545891 >> v1:172.21.15.121:6789/0 conn(0x7fd3a8143490 legacy=0x7fd3a80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/1163545891 >> v1:172.21.15.46:6789/0 conn(0x7fd3a815e730 legacy=0x7fd3a815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.208+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/1163545891 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a815dc50 con 0x7fd3a80b1400 2023-12-16T21:13:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.209+0000 7fd3a6ffd640 1 --2- 172.21.15.46:0/1163545891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a80b1400 0x7fd3a815a4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0097d0 tx=0x7fd39c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b46f8298b812d180 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.209+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c031890 con 0x7fd3a80b1400 2023-12-16T21:13:33.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.209+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c0319f0 con 0x7fd3a80b1400 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.209+0000 7fd3a67fc640 1 -- 172.21.15.46:0/1163545891 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c031d80 con 0x7fd3a80b1400 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 -- 172.21.15.46:0/1163545891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a80b1400 msgr2=0x7fd3a815a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 --2- 172.21.15.46:0/1163545891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a80b1400 0x7fd3a815a4e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0097d0 tx=0x7fd39c02b930 comp rx=0 tx=0).stop 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 -- 172.21.15.46:0/1163545891 shutdown_connections 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 --2- 172.21.15.46:0/1163545891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a80b1400 0x7fd3a815a4e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 -- 172.21.15.46:0/1163545891 >> 172.21.15.46:0/1163545891 conn(0x7fd3a806a4d0 msgr2=0x7fd3a806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 -- 172.21.15.46:0/1163545891 shutdown_connections 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.210+0000 7fd3ada72640 1 -- 172.21.15.46:0/1163545891 wait complete. 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.211+0000 7fd3ada72640 1 Processor -- start 2023-12-16T21:13:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.211+0000 7fd3ada72640 1 -- start start 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3ada72640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a80b1400 0x7fd3a8159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3ada72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 0x7fd3a80afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3ada72640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a815e730 0x7fd3a80b0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3ada72640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a813e150 con 0x7fd3a8143490 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3ada72640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a813d140 con 0x7fd3a80b1400 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3ada72640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a813d390 con 0x7fd3a815e730 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 0x7fd3a80afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 0x7fd3a80afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54272/0 (socket says 172.21.15.46:54272) 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/171230479 learned_addr learned my addr 172.21.15.46:0/171230479 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a77fe640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a80b1400 0x7fd3a8159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a7fff640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a815e730 0x7fd3a80b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a815e730 msgr2=0x7fd3a80b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a815e730 0x7fd3a80b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a80b1400 msgr2=0x7fd3a8159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a80b1400 0x7fd3a8159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 -- 172.21.15.46:0/171230479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c009400 con 0x7fd3a8143490 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a77fe640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a80b1400 0x7fd3a8159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a7fff640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a815e730 0x7fd3a80b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.212+0000 7fd3a6ffd640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 0x7fd3a80afd20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0098f0 tx=0x7fd39c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.213+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c03bbf0 con 0x7fd3a8143490 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.213+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c03bd50 con 0x7fd3a8143490 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.213+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a813d670 con 0x7fd3a8143490 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.213+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c045630 con 0x7fd3a8143490 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.213+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a813dc30 con 0x7fd3a8143490 2023-12-16T21:13:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.214+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a80b4380 con 0x7fd3a8143490 2023-12-16T21:13:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.214+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd39c04e450 con 0x7fd3a8143490 2023-12-16T21:13:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.214+0000 7fd38ffff640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39406ae40 0x7fd39406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.214+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd39c0ab8d0 con 0x7fd3a8143490 2023-12-16T21:13:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.215+0000 7fd3a77fe640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39406ae40 0x7fd39406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.215+0000 7fd3a77fe640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39406ae40 0x7fd39406d300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3980011e0 tx=0x7fd398009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.219+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd39c076cd0 con 0x7fd3a8143490 2023-12-16T21:13:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.393+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3a8069400 con 0x7fd3a8143490 2023-12-16T21:13:33.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.394+0000 7fd38ffff640 1 -- 172.21.15.46:0/171230479 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd39c035020 con 0x7fd3a8143490 2023-12-16T21:13:33.394 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:13:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39406ae40 msgr2=0x7fd39406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39406ae40 0x7fd39406d300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd3980011e0 tx=0x7fd398009040 comp rx=0 tx=0).stop 2023-12-16T21:13:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 msgr2=0x7fd3a80afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 0x7fd3a80afd20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd39c0098f0 tx=0x7fd39c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 shutdown_connections 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39406ae40 0x7fd39406d300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3a8143490 0x7fd3a80afd20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3a815e730 0x7fd3a80b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 --2- 172.21.15.46:0/171230479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a80b1400 0x7fd3a8159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 >> 172.21.15.46:0/171230479 conn(0x7fd3a806a4d0 msgr2=0x7fd3a80b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.395+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 shutdown_connections 2023-12-16T21:13:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.396+0000 7fd3ada72640 1 -- 172.21.15.46:0/171230479 wait complete. 2023-12-16T21:13:33.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-16T21:13:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.512+0000 7f4b446bb640 1 Processor -- start 2023-12-16T21:13:33.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.513+0000 7f4b446bb640 1 -- start start 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.513+0000 7f4b446bb640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b3c0b5370 0x7f4b3c0cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.513+0000 7f4b446bb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b3c0d2040 0x7f4b3c0cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.513+0000 7f4b446bb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0ce4c0 con 0x7f4b3c15d2d0 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.513+0000 7f4b446bb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0d1bb0 con 0x7f4b3c0d2040 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.513+0000 7f4b446bb640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c15d6b0 con 0x7f4b3c0b5370 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b3c0d2040 0x7f4b3c0cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b3c0d2040 0x7f4b3c0cdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54282/0 (socket says 172.21.15.46:54282) 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 -- 172.21.15.46:0/3563542287 learned_addr learned my addr 172.21.15.46:0/3563542287 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b41c56640 1 --2- 172.21.15.46:0/3563542287 >> v2:172.21.15.121:3300/0 conn(0x7f4b3c0b5370 0x7f4b3c0cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 -- 172.21.15.46:0/3563542287 >> v2:172.21.15.121:3300/0 conn(0x7f4b3c0b5370 msgr2=0x7f4b3c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 --2- 172.21.15.46:0/3563542287 >> v2:172.21.15.121:3300/0 conn(0x7f4b3c0b5370 0x7f4b3c0cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 -- 172.21.15.46:0/3563542287 >> v1:172.21.15.121:6789/0 conn(0x7f4b3c15d2d0 legacy=0x7f4b3c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.514+0000 7f4b42c58640 1 -- 172.21.15.46:0/3563542287 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0ce700 con 0x7f4b3c0d2040 2023-12-16T21:13:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b42c58640 1 --2- 172.21.15.46:0/3563542287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0d2040 0x7f4b3c0cdf80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0b2c10 tx=0x7f4b2c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a21af73ca856ffd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b41455640 1 -- 172.21.15.46:0/3563542287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c02cb20 con 0x7f4b3c0d2040 2023-12-16T21:13:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b41455640 1 -- 172.21.15.46:0/3563542287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2c02cc80 con 0x7f4b3c0d2040 2023-12-16T21:13:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b41455640 1 -- 172.21.15.46:0/3563542287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c0343f0 con 0x7f4b3c0d2040 2023-12-16T21:13:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 -- 172.21.15.46:0/3563542287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0d2040 msgr2=0x7f4b3c0cdf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3563542287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0d2040 0x7f4b3c0cdf80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0b2c10 tx=0x7f4b2c02bf80 comp rx=0 tx=0).stop 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 -- 172.21.15.46:0/3563542287 shutdown_connections 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3563542287 >> v2:172.21.15.121:3300/0 conn(0x7f4b3c0b5370 0x7f4b3c0cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3563542287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0d2040 0x7f4b3c0cdf80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 -- 172.21.15.46:0/3563542287 >> 172.21.15.46:0/3563542287 conn(0x7f4b3c072050 msgr2=0x7f4b3c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 -- 172.21.15.46:0/3563542287 shutdown_connections 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.515+0000 7f4b446bb640 1 -- 172.21.15.46:0/3563542287 wait complete. 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 Processor -- start 2023-12-16T21:13:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 -- start start 2023-12-16T21:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 0x7f4b3c0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c0d2040 0x7f4b3c0cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c15d2d0 0x7f4b3c1512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0c8690 con 0x7f4b3c0b5370 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0c75f0 con 0x7f4b3c15d2d0 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b446bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0c78d0 con 0x7f4b3c0d2040 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.517+0000 7f4b42457640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 0x7f4b3c0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 0x7f4b3c0c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54296/0 (socket says 172.21.15.46:54296) 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 -- 172.21.15.46:0/3254220526 learned_addr learned my addr 172.21.15.46:0/3254220526 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b41c56640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c0d2040 0x7f4b3c0cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42c58640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c15d2d0 0x7f4b3c1512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 -- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c0d2040 msgr2=0x7f4b3c0cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c0d2040 0x7f4b3c0cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 -- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c15d2d0 msgr2=0x7f4b3c1512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c15d2d0 0x7f4b3c1512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 -- 172.21.15.46:0/3254220526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c009b30 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42c58640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c15d2d0 0x7f4b3c1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b41c56640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c0d2040 0x7f4b3c0cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b42457640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 0x7f4b3c0c6cf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400c400 tx=0x7f4b3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340052d0 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b34005430 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c0c7bb0 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34003830 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.518+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c0c8170 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.519+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c069400 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.520+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4b34003990 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.520+0000 7f4b3affd640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2806ae40 0x7f4b2806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.520+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4b34088700 con 0x7f4b3c0b5370 2023-12-16T21:13:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.520+0000 7f4b41c56640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2806ae40 0x7f4b2806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.521+0000 7f4b41c56640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2806ae40 0x7f4b2806d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0a9cf0 tx=0x7f4b30008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.524+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4b34053b00 con 0x7f4b3c0b5370 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.701+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4b3c0ce4c0 con 0x7f4b3c0b5370 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.701+0000 7f4b3affd640 1 -- 172.21.15.46:0/3254220526 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4b34017060 con 0x7f4b3c0b5370 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:13:33.702 INFO:teuthology.orchestra.run.smithi046.stdout:] 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2806ae40 msgr2=0x7f4b2806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2806ae40 0x7f4b2806d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0a9cf0 tx=0x7f4b30008040 comp rx=0 tx=0).stop 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 msgr2=0x7f4b3c0c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 0x7f4b3c0c6cf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400c400 tx=0x7f4b3400c8c0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 shutdown_connections 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2806ae40 0x7f4b2806d300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c15d2d0 0x7f4b3c1512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c0d2040 0x7f4b3c0cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 --2- 172.21.15.46:0/3254220526 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c0b5370 0x7f4b3c0c6cf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 >> 172.21.15.46:0/3254220526 conn(0x7f4b3c072050 msgr2=0x7f4b3c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 shutdown_connections 2023-12-16T21:13:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.703+0000 7f4b446bb640 1 -- 172.21.15.46:0/3254220526 wait complete. 2023-12-16T21:13:33.714 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-16T21:13:33.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-16T21:13:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.823+0000 7f8c3bf2c640 1 Processor -- start 2023-12-16T21:13:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.824+0000 7f8c3bf2c640 1 -- start start 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.824+0000 7f8c3bf2c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d2910 0x7f8c340d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c3bf2c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340b2b10 con 0x7f8c340d3730 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c3bf2c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c340d3b50 con 0x7f8c3415d330 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c3bf2c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340d1590 con 0x7f8c340d2910 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c3a4c9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c340d3730 0x7f8c340b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33532/0 (socket says 172.21.15.46:33532) 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c3a4c9640 1 -- 172.21.15.46:0/4044738023 learned_addr learned my addr 172.21.15.46:0/4044738023 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c394c7640 1 --2- 172.21.15.46:0/4044738023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d2910 0x7f8c340d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584904033 0 0) 0x7f8c340b2b10 con 0x7f8c340d3730 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c20003520 con 0x7f8c340d3730 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344882706 0 0) 0x7f8c340d3b50 con 0x7f8c3415d330 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c340b2b10 con 0x7f8c3415d330 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 239228541 0 0) 0x7f8c20003520 con 0x7f8c340d3730 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c394c7640 1 -- 172.21.15.46:0/4044738023 >> v1:172.21.15.121:6789/0 conn(0x7f8c3415d330 legacy=0x7f8c340b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c394c7640 1 -- 172.21.15.46:0/4044738023 >> v1:172.21.15.46:6789/0 conn(0x7f8c340d3730 legacy=0x7f8c340b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.825+0000 7f8c394c7640 1 -- 172.21.15.46:0/4044738023 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c340d1870 con 0x7f8c340d2910 2023-12-16T21:13:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.826+0000 7f8c394c7640 1 --2- 172.21.15.46:0/4044738023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d2910 0x7f8c340d31f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c240097d0 tx=0x7f8c2402b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78fdd33e5c79cfeb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.826+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c0019f0 con 0x7f8c340d2910 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.826+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2402e050 con 0x7f8c340d2910 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.826+0000 7f8c38cc6640 1 -- 172.21.15.46:0/4044738023 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c24030730 con 0x7f8c340d2910 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/4044738023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d2910 msgr2=0x7f8c340d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/4044738023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d2910 0x7f8c340d31f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8c240097d0 tx=0x7f8c2402b770 comp rx=0 tx=0).stop 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/4044738023 shutdown_connections 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/4044738023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d2910 0x7f8c340d31f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/4044738023 >> 172.21.15.46:0/4044738023 conn(0x7f8c34072020 msgr2=0x7f8c34072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/4044738023 shutdown_connections 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.827+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/4044738023 wait complete. 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.828+0000 7f8c3bf2c640 1 Processor -- start 2023-12-16T21:13:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.828+0000 7f8c3bf2c640 1 -- start start 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3bf2c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 0x7f8c340c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3bf2c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d3730 0x7f8c340c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3bf2c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c3415d330 0x7f8c340c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3bf2c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3415bb80 con 0x7f8c340d2910 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3bf2c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c340cb380 con 0x7f8c3415d330 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3bf2c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3415bdd0 con 0x7f8c340d3730 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 0x7f8c340c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 0x7f8c340c7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54304/0 (socket says 172.21.15.46:54304) 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 -- 172.21.15.46:0/2139221965 learned_addr learned my addr 172.21.15.46:0/2139221965 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c394c7640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d3730 0x7f8c340c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c3a4c9640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c3415d330 0x7f8c340c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 -- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d3730 msgr2=0x7f8c340c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d3730 0x7f8c340c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 -- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c3415d330 msgr2=0x7f8c340c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c3415d330 0x7f8c340c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 -- 172.21.15.46:0/2139221965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24009400 con 0x7f8c340d2910 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.829+0000 7f8c39cc8640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 0x7f8c340c7560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000c430 tx=0x7f8c3000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.830+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c300052d0 con 0x7f8c340d2910 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.830+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c30005450 con 0x7f8c340d2910 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.830+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3415c0b0 con 0x7f8c340d2910 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.830+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30003830 con 0x7f8c340d2910 2023-12-16T21:13:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.830+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c340cb8b0 con 0x7f8c340d2910 2023-12-16T21:13:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.831+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3415c2f0 con 0x7f8c340d2910 2023-12-16T21:13:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.831+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8c30003990 con 0x7f8c340d2910 2023-12-16T21:13:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.831+0000 7f8c2a7fc640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1806ae40 0x7f8c1806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.831+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8c300889b0 con 0x7f8c340d2910 2023-12-16T21:13:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.832+0000 7f8c394c7640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1806ae40 0x7f8c1806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.832+0000 7f8c394c7640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1806ae40 0x7f8c1806d300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c24034000 tx=0x7f8c24008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:33.835+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8c30053d20 con 0x7f8c340d2910 2023-12-16T21:13:34.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.011+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8c340cbaa0 con 0x7f8c340d2910 2023-12-16T21:13:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.485+0000 7f8c2a7fc640 1 -- 172.21.15.46:0/2139221965 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8c30057340 con 0x7f8c340d2910 2023-12-16T21:13:34.486 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2023-12-16T21:13:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1806ae40 msgr2=0x7f8c1806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1806ae40 0x7f8c1806d300 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8c24034000 tx=0x7f8c24008030 comp rx=0 tx=0).stop 2023-12-16T21:13:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 msgr2=0x7f8c340c7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 0x7f8c340c7560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c3000c430 tx=0x7f8c3000c8f0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 shutdown_connections 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1806ae40 0x7f8c1806d300 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c3415d330 0x7f8c340c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c340d3730 0x7f8c340c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 --2- 172.21.15.46:0/2139221965 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c340d2910 0x7f8c340c7560 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.491+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 >> 172.21.15.46:0/2139221965 conn(0x7f8c34072020 msgr2=0x7f8c340b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.492+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 shutdown_connections 2023-12-16T21:13:34.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.492+0000 7f8c3bf2c640 1 -- 172.21.15.46:0/2139221965 wait complete. 2023-12-16T21:13:34.506 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-16T21:13:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.622+0000 7f7b870fc640 1 Processor -- start 2023-12-16T21:13:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.622+0000 7f7b870fc640 1 -- start start 2023-12-16T21:13:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.622+0000 7f7b870fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b8006f9c0 0x7f7b800cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.622+0000 7f7b870fc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b80147150 con 0x7f7b800cf490 2023-12-16T21:13:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.622+0000 7f7b870fc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800cfd80 con 0x7f7b800b2180 2023-12-16T21:13:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.622+0000 7f7b870fc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8015e570 con 0x7f7b8006f9c0 2023-12-16T21:13:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b8006f9c0 0x7f7b800cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b85699640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7b800cf490 0x7f7b801610b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33546/0 (socket says 172.21.15.46:33546) 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b7ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7b8006f9c0 0x7f7b800cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54318/0 (socket says 172.21.15.46:54318) 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b85699640 1 -- 172.21.15.46:0/2010187742 learned_addr learned my addr 172.21.15.46:0/2010187742 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b7ffff640 1 -- 172.21.15.46:0/2010187742 >> v1:172.21.15.121:6789/0 conn(0x7f7b800b2180 legacy=0x7f7b8006f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b7ffff640 1 -- 172.21.15.46:0/2010187742 >> v1:172.21.15.46:6789/0 conn(0x7f7b800cf490 legacy=0x7f7b801610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.623+0000 7f7b7ffff640 1 -- 172.21.15.46:0/2010187742 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8015dbd0 con 0x7f7b8006f9c0 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b7ffff640 1 --2- 172.21.15.46:0/2010187742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b8006f9c0 0x7f7b800cef50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b800cff70 tx=0x7f7b6c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280137732fba8c93 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b7f7fe640 1 -- 172.21.15.46:0/2010187742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c030610 con 0x7f7b8006f9c0 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b7f7fe640 1 -- 172.21.15.46:0/2010187742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c030770 con 0x7f7b8006f9c0 2023-12-16T21:13:34.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b7f7fe640 1 -- 172.21.15.46:0/2010187742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c030b00 con 0x7f7b8006f9c0 2023-12-16T21:13:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 -- 172.21.15.46:0/2010187742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b8006f9c0 msgr2=0x7f7b800cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 --2- 172.21.15.46:0/2010187742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b8006f9c0 0x7f7b800cef50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7b800cff70 tx=0x7f7b6c02b960 comp rx=0 tx=0).stop 2023-12-16T21:13:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 -- 172.21.15.46:0/2010187742 shutdown_connections 2023-12-16T21:13:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 --2- 172.21.15.46:0/2010187742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b8006f9c0 0x7f7b800cef50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 -- 172.21.15.46:0/2010187742 >> 172.21.15.46:0/2010187742 conn(0x7f7b8006d690 msgr2=0x7f7b8006da70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:34.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 -- 172.21.15.46:0/2010187742 shutdown_connections 2023-12-16T21:13:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.624+0000 7f7b870fc640 1 -- 172.21.15.46:0/2010187742 wait complete. 2023-12-16T21:13:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 Processor -- start 2023-12-16T21:13:34.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 -- start start 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8006f9c0 0x7f7b8015d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b800b2180 0x7f7b8015be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 0x7f7b800aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8013b410 con 0x7f7b800cf490 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8015d0c0 con 0x7f7b800b2180 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.626+0000 7f7b870fc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b80159f30 con 0x7f7b8006f9c0 2023-12-16T21:13:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 0x7f7b800aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 0x7f7b800aeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54328/0 (socket says 172.21.15.46:54328) 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 -- 172.21.15.46:0/332339808 learned_addr learned my addr 172.21.15.46:0/332339808 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b7ffff640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b800b2180 0x7f7b8015be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b84e98640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8006f9c0 0x7f7b8015d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 -- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8006f9c0 msgr2=0x7f7b8015d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8006f9c0 0x7f7b8015d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 -- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b800b2180 msgr2=0x7f7b8015be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b800b2180 0x7f7b8015be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 -- 172.21.15.46:0/332339808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c009400 con 0x7f7b800cf490 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b7ffff640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b800b2180 0x7f7b8015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b85699640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 0x7f7b800aeec0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800cbf0 tx=0x7f7b7800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7800ad40 con 0x7f7b800cf490 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b78012400 con 0x7f7b800cf490 2023-12-16T21:13:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8015c4b0 con 0x7f7b800cf490 2023-12-16T21:13:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7800aea0 con 0x7f7b800cf490 2023-12-16T21:13:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.627+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8015ca70 con 0x7f7b800cf490 2023-12-16T21:13:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.628+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b80069400 con 0x7f7b800cf490 2023-12-16T21:13:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.629+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7b78007070 con 0x7f7b800cf490 2023-12-16T21:13:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.629+0000 7f7b7d7fa640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7b6406ae40 0x7f7b6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.629+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7b78089210 con 0x7f7b800cf490 2023-12-16T21:13:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.629+0000 7f7b84e98640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7b6406ae40 0x7f7b6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.630+0000 7f7b84e98640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7b6406ae40 0x7f7b6406d300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7b74000f60 tx=0x7f7b74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.634+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7b780545e0 con 0x7f7b800cf490 2023-12-16T21:13:34.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:34.812+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7b8015d200 con 0x7f7b800cf490 2023-12-16T21:13:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.488+0000 7f7b7d7fa640 1 -- 172.21.15.46:0/332339808 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7b78057c00 con 0x7f7b800cf490 2023-12-16T21:13:35.488 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7b6406ae40 msgr2=0x7f7b6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7b6406ae40 0x7f7b6406d300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7b74000f60 tx=0x7f7b74009040 comp rx=0 tx=0).stop 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 msgr2=0x7f7b800aeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 0x7f7b800aeec0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800cbf0 tx=0x7f7b7800a430 comp rx=0 tx=0).stop 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 shutdown_connections 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7b6406ae40 0x7f7b6406d300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7b800cf490 0x7f7b800aeec0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b800b2180 0x7f7b8015be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 --2- 172.21.15.46:0/332339808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b8006f9c0 0x7f7b8015d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.491+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 >> 172.21.15.46:0/332339808 conn(0x7f7b8006d690 msgr2=0x7f7b800b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.492+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 shutdown_connections 2023-12-16T21:13:35.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.492+0000 7f7b870fc640 1 -- 172.21.15.46:0/332339808 wait complete. 2023-12-16T21:13:35.517 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-16T21:13:35.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.631+0000 7f9cbc615640 1 Processor -- start 2023-12-16T21:13:35.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.631+0000 7f9cbc615640 1 -- start start 2023-12-16T21:13:35.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbc615640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbc615640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cb40709c0 0x7f9cb40cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbc615640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb40d0c40 con 0x7f9cb40cdd10 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbc615640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb40ce1a0 con 0x7f9cb40709c0 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbc615640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40ced90 con 0x7f9cb415d270 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cb9bb0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cb40709c0 0x7f9cb40cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbabb2640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9cb40cdd10 0x7f9cb40cc6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33562/0 (socket says 172.21.15.46:33562) 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cb9bb0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9cb40709c0 0x7f9cb40cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54338/0 (socket says 172.21.15.46:54338) 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.632+0000 7f9cbabb2640 1 -- 172.21.15.46:0/4072409588 learned_addr learned my addr 172.21.15.46:0/4072409588 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.633+0000 7f9cba3b1640 1 --2- 172.21.15.46:0/4072409588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.633+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/4072409588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 msgr2=0x7f9cb406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.633+0000 7f9cb9bb0640 1 --2- 172.21.15.46:0/4072409588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.633+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/4072409588 >> v1:172.21.15.46:6789/0 conn(0x7f9cb40cdd10 legacy=0x7f9cb40cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:35.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.633+0000 7f9cb93af640 1 -- 172.21.15.46:0/4072409588 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567916460 0 0) 0x7f9cb40d0c40 con 0x7f9cb40cdd10 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.633+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/4072409588 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40cf070 con 0x7f9cb40709c0 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cb9bb0640 1 --2- 172.21.15.46:0/4072409588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40709c0 0x7f9cb40cd7d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80097a0 tx=0x7f9ca802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ec8da5fae0ae46 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cb93af640 1 -- 172.21.15.46:0/4072409588 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca802e070 con 0x7f9cb40709c0 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cb93af640 1 -- 172.21.15.46:0/4072409588 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca80315a0 con 0x7f9cb40709c0 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cb93af640 1 -- 172.21.15.46:0/4072409588 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8031930 con 0x7f9cb40709c0 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 -- 172.21.15.46:0/4072409588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40709c0 msgr2=0x7f9cb40cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 --2- 172.21.15.46:0/4072409588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40709c0 0x7f9cb40cd7d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80097a0 tx=0x7f9ca802b840 comp rx=0 tx=0).stop 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 -- 172.21.15.46:0/4072409588 shutdown_connections 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 --2- 172.21.15.46:0/4072409588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40709c0 0x7f9cb40cd7d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 --2- 172.21.15.46:0/4072409588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb406ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 -- 172.21.15.46:0/4072409588 >> 172.21.15.46:0/4072409588 conn(0x7f9cb40b2c10 msgr2=0x7f9cb40b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 -- 172.21.15.46:0/4072409588 shutdown_connections 2023-12-16T21:13:35.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.634+0000 7f9cbc615640 1 -- 172.21.15.46:0/4072409588 wait complete. 2023-12-16T21:13:35.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 Processor -- start 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 -- start start 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cb40709c0 0x7f9cb414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 0x7f9cb41e8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb41e9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c8e00 con 0x7f9cb40cdd10 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c90e0 con 0x7f9cb40709c0 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cbc615640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c93c0 con 0x7f9cb415d270 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cb9bb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 0x7f9cb41e8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cb9bb0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 0x7f9cb41e8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54342/0 (socket says 172.21.15.46:54342) 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.636+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/3878441247 learned_addr learned my addr 172.21.15.46:0/3878441247 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cbabb2640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb41e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cba3b1640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cb40709c0 0x7f9cb414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:35.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 msgr2=0x7f9cb41e9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb9bb0640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb41e9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cb40709c0 msgr2=0x7f9cb414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb9bb0640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cb40709c0 0x7f9cb414f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb9bb0640 1 -- 172.21.15.46:0/3878441247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca8009400 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb9bb0640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 0x7f9cb41e8ec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80097a0 tx=0x7f9ca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8004230 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca8031f00 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb40c96a0 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8032640 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.637+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb414f970 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.638+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb4067fc0 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.639+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9ca80327a0 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.639+0000 7f9cb2ffd640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ca006ae40 0x7f9ca006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.639+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f9ca80ab700 con 0x7f9cb40cdd10 2023-12-16T21:13:35.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.639+0000 7f9cba3b1640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ca006ae40 0x7f9ca006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.640+0000 7f9cba3b1640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ca006ae40 0x7f9ca006d300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009a60 tx=0x7f9ca4003990 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:35.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.645+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9ca8076780 con 0x7f9cb40cdd10 2023-12-16T21:13:35.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:35.858+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9cb40c9bc0 con 0x7f9cb40cdd10 2023-12-16T21:13:36.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.511+0000 7f9cb2ffd640 1 -- 172.21.15.46:0/3878441247 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9ca8079da0 con 0x7f9cb40cdd10 2023-12-16T21:13:36.511 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2023-12-16T21:13:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ca006ae40 msgr2=0x7f9ca006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ca006ae40 0x7f9ca006d300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009a60 tx=0x7f9ca4003990 comp rx=0 tx=0).stop 2023-12-16T21:13:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 msgr2=0x7f9cb41e8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 0x7f9cb41e8ec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ca80097a0 tx=0x7f9ca80046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 shutdown_connections 2023-12-16T21:13:36.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ca006ae40 0x7f9ca006d300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9cb40cdd10 0x7f9cb41e8ec0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cb415d270 0x7f9cb41e9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 --2- 172.21.15.46:0/3878441247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cb40709c0 0x7f9cb414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 >> 172.21.15.46:0/3878441247 conn(0x7f9cb40b2c10 msgr2=0x7f9cb40b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 shutdown_connections 2023-12-16T21:13:36.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.513+0000 7f9cbc615640 1 -- 172.21.15.46:0/3878441247 wait complete. 2023-12-16T21:13:36.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:36.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.641+0000 7fee23b16640 1 Processor -- start 2023-12-16T21:13:36.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.641+0000 7fee23b16640 1 -- start start 2023-12-16T21:13:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee23b16640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee23b16640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fee1c0ce480 0x7fee1c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee23b16640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c1614e0 con 0x7fee1c15d2d0 2023-12-16T21:13:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee23b16640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c1617c0 con 0x7fee1c0ce480 2023-12-16T21:13:36.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee23b16640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c161aa0 con 0x7fee1c160f70 2023-12-16T21:13:36.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:47888/0 (socket says 172.21.15.46:47888) 2023-12-16T21:13:36.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 -- 172.21.15.46:0/1379020721 learned_addr learned my addr 172.21.15.46:0/1379020721 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:36.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee220b3640 1 --2- 172.21.15.46:0/1379020721 >> v2:172.21.15.121:3300/0 conn(0x7fee1c0ce480 0x7fee1c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 -- 172.21.15.46:0/1379020721 >> v2:172.21.15.121:3300/0 conn(0x7fee1c0ce480 msgr2=0x7fee1c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:36.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 --2- 172.21.15.46:0/1379020721 >> v2:172.21.15.121:3300/0 conn(0x7fee1c0ce480 0x7fee1c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 -- 172.21.15.46:0/1379020721 >> v1:172.21.15.121:6789/0 conn(0x7fee1c15d2d0 legacy=0x7fee1c0b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.642+0000 7fee210b1640 1 -- 172.21.15.46:0/1379020721 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c161d80 con 0x7fee1c160f70 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.643+0000 7fee210b1640 1 --2- 172.21.15.46:0/1379020721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0cdf40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009870 tx=0x7fee0c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2e4700e9b629461 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.643+0000 7fee208b0640 1 -- 172.21.15.46:0/1379020721 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c030840 con 0x7fee1c160f70 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.643+0000 7fee208b0640 1 -- 172.21.15.46:0/1379020721 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c0309a0 con 0x7fee1c160f70 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.643+0000 7fee208b0640 1 -- 172.21.15.46:0/1379020721 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c030d30 con 0x7fee1c160f70 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 -- 172.21.15.46:0/1379020721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 msgr2=0x7fee1c0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 --2- 172.21.15.46:0/1379020721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0cdf40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009870 tx=0x7fee0c02b980 comp rx=0 tx=0).stop 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 -- 172.21.15.46:0/1379020721 shutdown_connections 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 --2- 172.21.15.46:0/1379020721 >> v2:172.21.15.121:3300/0 conn(0x7fee1c0ce480 0x7fee1c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 --2- 172.21.15.46:0/1379020721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0cdf40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 -- 172.21.15.46:0/1379020721 >> 172.21.15.46:0/1379020721 conn(0x7fee1c072230 msgr2=0x7fee1c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 -- 172.21.15.46:0/1379020721 shutdown_connections 2023-12-16T21:13:36.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.644+0000 7fee23b16640 1 -- 172.21.15.46:0/1379020721 wait complete. 2023-12-16T21:13:36.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.645+0000 7fee23b16640 1 Processor -- start 2023-12-16T21:13:36.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.645+0000 7fee23b16640 1 -- start start 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee23b16640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee1c0ce480 0x7fee1c0c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee23b16640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 0x7fee1c157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee23b16640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee23b16640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c14f740 con 0x7fee1c15d2d0 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee23b16640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c15a5f0 con 0x7fee1c0ce480 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee23b16640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c1599e0 con 0x7fee1c160f70 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 0x7fee1c157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 0x7fee1c157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54348/0 (socket says 172.21.15.46:54348) 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 -- 172.21.15.46:0/1145973306 learned_addr learned my addr 172.21.15.46:0/1145973306 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee218b2640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee1c0ce480 0x7fee1c0c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee220b3640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 -- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 msgr2=0x7fee1c0c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 -- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee1c0ce480 msgr2=0x7fee1c0c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee1c0ce480 0x7fee1c0c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 -- 172.21.15.46:0/1145973306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0094d0 con 0x7fee1c15d2d0 2023-12-16T21:13:36.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.646+0000 7fee210b1640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 0x7fee1c157970 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0090b0 tx=0x7fee0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:36.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.647+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c03e070 con 0x7fee1c15d2d0 2023-12-16T21:13:36.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.647+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0c039a10 con 0x7fee1c15d2d0 2023-12-16T21:13:36.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.647+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c159cc0 con 0x7fee1c15d2d0 2023-12-16T21:13:36.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.647+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c039de0 con 0x7fee1c15d2d0 2023-12-16T21:13:36.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.647+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c15a280 con 0x7fee1c15d2d0 2023-12-16T21:13:36.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.648+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c13f330 con 0x7fee1c15d2d0 2023-12-16T21:13:36.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.648+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fee0c04b400 con 0x7fee1c15d2d0 2023-12-16T21:13:36.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.649+0000 7fee127fc640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fee0006ae40 0x7fee0006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.649+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee0c0aba30 con 0x7fee1c15d2d0 2023-12-16T21:13:36.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.649+0000 7fee218b2640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fee0006ae40 0x7fee0006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.649+0000 7fee218b2640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fee0006ae40 0x7fee0006d300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee18000f60 tx=0x7fee18009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:36.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.653+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fee1c13f330 con 0x7fee1c15d2d0 2023-12-16T21:13:36.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.831+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee1c13f330 con 0x7fee1c15d2d0 2023-12-16T21:13:36.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.832+0000 7fee127fc640 1 -- 172.21.15.46:0/1145973306 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16518 (secure 0 0 0) 0x7fee1c13f330 con 0x7fee1c15d2d0 2023-12-16T21:13:36.832 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:36.832 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:35.860831+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fee0006ae40 msgr2=0x7fee0006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fee0006ae40 0x7fee0006d300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee18000f60 tx=0x7fee18009040 comp rx=0 tx=0).stop 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 msgr2=0x7fee1c157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 0x7fee1c157970 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0090b0 tx=0x7fee0c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 shutdown_connections 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fee0006ae40 0x7fee0006d300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fee1c15d2d0 0x7fee1c157970 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fee1c160f70 0x7fee1c0c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 --2- 172.21.15.46:0/1145973306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee1c0ce480 0x7fee1c0c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 >> 172.21.15.46:0/1145973306 conn(0x7fee1c072230 msgr2=0x7fee1c0b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:36.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 shutdown_connections 2023-12-16T21:13:36.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.834+0000 7fee23b16640 1 -- 172.21.15.46:0/1145973306 wait complete. 2023-12-16T21:13:36.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:36.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.951+0000 7fc1e643d640 1 Processor -- start 2023-12-16T21:13:36.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.951+0000 7fc1e643d640 1 -- start start 2023-12-16T21:13:36.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1e643d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e015c2f0 0x7fc1e006eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1e643d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1e006f040 0x7fc1e0140800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1e643d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1e015ba40 con 0x7fc1e0140d40 2023-12-16T21:13:36.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1e643d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1e015c6d0 con 0x7fc1e006f040 2023-12-16T21:13:36.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1e643d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e013e530 con 0x7fc1e015c2f0 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1e006f040 0x7fc1e0140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1e006f040 0x7fc1e0140800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54360/0 (socket says 172.21.15.46:54360) 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 -- 172.21.15.46:0/1949638779 learned_addr learned my addr 172.21.15.46:0/1949638779 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1dffff640 1 --2- 172.21.15.46:0/1949638779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e015c2f0 0x7fc1e006eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 -- 172.21.15.46:0/1949638779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e015c2f0 msgr2=0x7fc1e006eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 --2- 172.21.15.46:0/1949638779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e015c2f0 0x7fc1e006eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 -- 172.21.15.46:0/1949638779 >> v1:172.21.15.121:6789/0 conn(0x7fc1e0140d40 legacy=0x7fc1e015ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:36.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.952+0000 7fc1df7fe640 1 -- 172.21.15.46:0/1949638779 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e01411d0 con 0x7fc1e006f040 2023-12-16T21:13:36.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.953+0000 7fc1df7fe640 1 --2- 172.21.15.46:0/1949638779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e006f040 0x7fc1e0140800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00098d0 tx=0x7fc1d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c305a916478303 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:36.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.953+0000 7fc1deffd640 1 -- 172.21.15.46:0/1949638779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0031830 con 0x7fc1e006f040 2023-12-16T21:13:36.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.953+0000 7fc1deffd640 1 -- 172.21.15.46:0/1949638779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d0031990 con 0x7fc1e006f040 2023-12-16T21:13:36.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.953+0000 7fc1deffd640 1 -- 172.21.15.46:0/1949638779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d0031d20 con 0x7fc1e006f040 2023-12-16T21:13:36.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 -- 172.21.15.46:0/1949638779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e006f040 msgr2=0x7fc1e0140800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 --2- 172.21.15.46:0/1949638779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e006f040 0x7fc1e0140800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc1d00098d0 tx=0x7fc1d002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 -- 172.21.15.46:0/1949638779 shutdown_connections 2023-12-16T21:13:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 --2- 172.21.15.46:0/1949638779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e006f040 0x7fc1e0140800 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 --2- 172.21.15.46:0/1949638779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e015c2f0 0x7fc1e006eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 -- 172.21.15.46:0/1949638779 >> 172.21.15.46:0/1949638779 conn(0x7fc1e006a6b0 msgr2=0x7fc1e006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 -- 172.21.15.46:0/1949638779 shutdown_connections 2023-12-16T21:13:36.955 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.954+0000 7fc1e643d640 1 -- 172.21.15.46:0/1949638779 wait complete. 2023-12-16T21:13:36.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.955+0000 7fc1e643d640 1 Processor -- start 2023-12-16T21:13:36.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.955+0000 7fc1e643d640 1 -- start start 2023-12-16T21:13:36.957 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e643d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e006f040 0x7fc1e01505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e643d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0140d40 0x7fc1e01e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e643d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 0x7fc1e01e5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e643d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01e6560 con 0x7fc1e015c2f0 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e643d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01e6840 con 0x7fc1e006f040 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e643d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e01e6b20 con 0x7fc1e0140d40 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 0x7fc1e01e5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 0x7fc1e01e5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54364/0 (socket says 172.21.15.46:54364) 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 -- 172.21.15.46:0/508082292 learned_addr learned my addr 172.21.15.46:0/508082292 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1df7fe640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0140d40 0x7fc1e01e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1dffff640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e006f040 0x7fc1e01505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 -- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0140d40 msgr2=0x7fc1e01e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0140d40 0x7fc1e01e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 -- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e006f040 msgr2=0x7fc1e01505f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.956+0000 7fc1e49da640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e006f040 0x7fc1e01505f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1e49da640 1 -- 172.21.15.46:0/508082292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d00094d0 con 0x7fc1e015c2f0 2023-12-16T21:13:36.958 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1df7fe640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0140d40 0x7fc1e01e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1e49da640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 0x7fc1e01e5ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c780 tx=0x7fc1d800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8004b10 con 0x7fc1e015c2f0 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1d8004c70 con 0x7fc1e015c2f0 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e006f4a0 con 0x7fc1e015c2f0 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d800aa90 con 0x7fc1e015c2f0 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.957+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e01e70e0 con 0x7fc1e015c2f0 2023-12-16T21:13:36.959 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.958+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e013e810 con 0x7fc1e015c2f0 2023-12-16T21:13:36.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.958+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc1d800ac10 con 0x7fc1e015c2f0 2023-12-16T21:13:36.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.959+0000 7fc1dcff9640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1cc06ae40 0x7fc1cc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:36.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.959+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1d80899b0 con 0x7fc1e015c2f0 2023-12-16T21:13:36.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.959+0000 7fc1dffff640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1cc06ae40 0x7fc1cc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:36.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.959+0000 7fc1dffff640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1cc06ae40 0x7fc1cc06d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4006fd0 tx=0x7fc1d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:36.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:36.963+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc1d8054a60 con 0x7fc1e015c2f0 2023-12-16T21:13:37.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.139+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1e0069400 con 0x7fc1e015c2f0 2023-12-16T21:13:37.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.140+0000 7fc1dcff9640 1 -- 172.21.15.46:0/508082292 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16518 (secure 0 0 0) 0x7fc1d8058080 con 0x7fc1e015c2f0 2023-12-16T21:13:37.140 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:37.140 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:35.860831+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1cc06ae40 msgr2=0x7fc1cc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1cc06ae40 0x7fc1cc06d300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc1d4006fd0 tx=0x7fc1d4008040 comp rx=0 tx=0).stop 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 msgr2=0x7fc1e01e5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 0x7fc1e01e5ee0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1d800c780 tx=0x7fc1d800cc40 comp rx=0 tx=0).stop 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 shutdown_connections 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1cc06ae40 0x7fc1cc06d300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1e015c2f0 0x7fc1e01e5ee0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0140d40 0x7fc1e01e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 --2- 172.21.15.46:0/508082292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e006f040 0x7fc1e01505f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 >> 172.21.15.46:0/508082292 conn(0x7fc1e006a6b0 msgr2=0x7fc1e006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:37.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 shutdown_connections 2023-12-16T21:13:37.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.142+0000 7fc1e643d640 1 -- 172.21.15.46:0/508082292 wait complete. 2023-12-16T21:13:37.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-16T21:13:37.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.264+0000 7fdd1afdc640 1 Processor -- start 2023-12-16T21:13:37.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.264+0000 7fdd1afdc640 1 -- start start 2023-12-16T21:13:37.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd1afdc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd1415d2d0 0x7fdd140b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd1afdc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd140ce150 0x7fdd140b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd1afdc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd140d1bb0 con 0x7fdd140d2040 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd1afdc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd1415d6b0 con 0x7fdd1415d2d0 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd1afdc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd140ce700 con 0x7fdd140ce150 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd18d78640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd1415d2d0 0x7fdd140b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd13fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fdd140d2040 0x7fdd140cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33578/0 (socket says 172.21.15.46:33578) 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd13fff640 1 -- 172.21.15.46:0/1807969332 learned_addr learned my addr 172.21.15.46:0/1807969332 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd19579640 1 --2- 172.21.15.46:0/1807969332 >> v2:172.21.15.121:3300/0 conn(0x7fdd140ce150 0x7fdd140b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd18d78640 1 -- 172.21.15.46:0/1807969332 >> v2:172.21.15.121:3300/0 conn(0x7fdd140ce150 msgr2=0x7fdd140b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd18d78640 1 --2- 172.21.15.46:0/1807969332 >> v2:172.21.15.121:3300/0 conn(0x7fdd140ce150 0x7fdd140b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd18d78640 1 -- 172.21.15.46:0/1807969332 >> v1:172.21.15.46:6789/0 conn(0x7fdd140d2040 legacy=0x7fdd140cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd18d78640 1 -- 172.21.15.46:0/1807969332 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd140d1280 con 0x7fdd1415d2d0 2023-12-16T21:13:37.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd137fe640 1 -- 172.21.15.46:0/1807969332 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568283590 0 0) 0x7fdd140d1bb0 con 0x7fdd140d2040 2023-12-16T21:13:37.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.265+0000 7fdd19579640 1 --2- 172.21.15.46:0/1807969332 >> v2:172.21.15.121:3300/0 conn(0x7fdd140ce150 0x7fdd140b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:37.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.266+0000 7fdd18d78640 1 --2- 172.21.15.46:0/1807969332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140b4c60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd140cd7c0 tx=0x7fdd0802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d740a3fbcc9cc821 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:37.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.266+0000 7fdd137fe640 1 -- 172.21.15.46:0/1807969332 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08031830 con 0x7fdd1415d2d0 2023-12-16T21:13:37.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.266+0000 7fdd137fe640 1 -- 172.21.15.46:0/1807969332 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd08031990 con 0x7fdd1415d2d0 2023-12-16T21:13:37.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.266+0000 7fdd137fe640 1 -- 172.21.15.46:0/1807969332 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08031d20 con 0x7fdd1415d2d0 2023-12-16T21:13:37.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 -- 172.21.15.46:0/1807969332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 msgr2=0x7fdd140b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/1807969332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140b4c60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd140cd7c0 tx=0x7fdd0802b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 -- 172.21.15.46:0/1807969332 shutdown_connections 2023-12-16T21:13:37.269 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/1807969332 >> v2:172.21.15.121:3300/0 conn(0x7fdd140ce150 0x7fdd140b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/1807969332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140b4c60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 -- 172.21.15.46:0/1807969332 >> 172.21.15.46:0/1807969332 conn(0x7fdd14072050 msgr2=0x7fdd14072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 -- 172.21.15.46:0/1807969332 shutdown_connections 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.267+0000 7fdd1afdc640 1 -- 172.21.15.46:0/1807969332 wait complete. 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.268+0000 7fdd1afdc640 1 Processor -- start 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.268+0000 7fdd1afdc640 1 -- start start 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd140ce150 0x7fdd14158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140d2040 0x7fdd140cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140ca860 con 0x7fdd1415d2d0 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140cab40 con 0x7fdd140d2040 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd140cae20 con 0x7fdd140ce150 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140cbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54390/0 (socket says 172.21.15.46:54390) 2023-12-16T21:13:37.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 -- 172.21.15.46:0/3635424012 learned_addr learned my addr 172.21.15.46:0/3635424012 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:37.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd13fff640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140d2040 0x7fdd140cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd18d78640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd140ce150 0x7fdd14158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 -- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd140ce150 msgr2=0x7fdd14158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd140ce150 0x7fdd14158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 -- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140d2040 msgr2=0x7fdd140cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140d2040 0x7fdd140cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 -- 172.21.15.46:0/3635424012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08009400 con 0x7fdd1415d2d0 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd18d78640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd140ce150 0x7fdd14158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:37.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd19579640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140cbe40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00cef0 tx=0x7fdd0c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:37.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00d840 con 0x7fdd1415d2d0 2023-12-16T21:13:37.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0c00d9c0 con 0x7fdd1415d2d0 2023-12-16T21:13:37.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd140cb100 con 0x7fdd1415d2d0 2023-12-16T21:13:37.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00dd70 con 0x7fdd1415d2d0 2023-12-16T21:13:37.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.269+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd141591e0 con 0x7fdd1415d2d0 2023-12-16T21:13:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.270+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd140cb620 con 0x7fdd1415d2d0 2023-12-16T21:13:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.271+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdd0c01e3b0 con 0x7fdd1415d2d0 2023-12-16T21:13:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.271+0000 7fdd117fa640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdcf806ae40 0x7fdcf806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.271+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd0c089f80 con 0x7fdd1415d2d0 2023-12-16T21:13:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.272+0000 7fdd18d78640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdcf806ae40 0x7fdcf806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.272+0000 7fdd18d78640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdcf806ae40 0x7fdcf806d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd08036000 tx=0x7fdd0803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:37.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.276+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdd0c0bcea0 con 0x7fdd1415d2d0 2023-12-16T21:13:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.493+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdd14069400 con 0x7fdd1415d2d0 2023-12-16T21:13:37.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.515+0000 7fdd117fa640 1 -- 172.21.15.46:0/3635424012 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdd0c01a0b0 con 0x7fdd1415d2d0 2023-12-16T21:13:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.517+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdcf806ae40 msgr2=0x7fdcf806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.517+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdcf806ae40 0x7fdcf806d300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdd08036000 tx=0x7fdd0803c040 comp rx=0 tx=0).stop 2023-12-16T21:13:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.517+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 msgr2=0x7fdd140cbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.517+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140cbe40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00cef0 tx=0x7fdd0c00a710 comp rx=0 tx=0).stop 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 shutdown_connections 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdcf806ae40 0x7fdcf806d300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd1415d2d0 0x7fdd140cbe40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140d2040 0x7fdd140cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 --2- 172.21.15.46:0/3635424012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd140ce150 0x7fdd14158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 >> 172.21.15.46:0/3635424012 conn(0x7fdd14072050 msgr2=0x7fdd140b3530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 shutdown_connections 2023-12-16T21:13:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.518+0000 7fdd1afdc640 1 -- 172.21.15.46:0/3635424012 wait complete. 2023-12-16T21:13:37.527 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-16T21:13:37.528 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-16T21:13:37.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.637+0000 7fb55cf1d640 1 Processor -- start 2023-12-16T21:13:37.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.637+0000 7fb55cf1d640 1 -- start start 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb55cf1d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb55815d2e0 0x7fb5580d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb55cf1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb55806bbd0 0x7fb5580d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb55cf1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806ce00 con 0x7fb55815d6c0 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb55cf1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55806c1f0 con 0x7fb55806bbd0 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb55cf1d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55806c4d0 con 0x7fb55815d2e0 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb556d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb55806bbd0 0x7fb5580d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb556d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb55806bbd0 0x7fb5580d1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54404/0 (socket says 172.21.15.46:54404) 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb555d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb55815d6c0 0x7fb55806b4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33588/0 (socket says 172.21.15.46:33588) 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.638+0000 7fb556d9d640 1 -- 172.21.15.46:0/3241049532 learned_addr learned my addr 172.21.15.46:0/3241049532 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb55659c640 1 --2- 172.21.15.46:0/3241049532 >> v2:172.21.15.121:3300/0 conn(0x7fb55815d2e0 0x7fb5580d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb556d9d640 1 -- 172.21.15.46:0/3241049532 >> v2:172.21.15.121:3300/0 conn(0x7fb55815d2e0 msgr2=0x7fb5580d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb556d9d640 1 --2- 172.21.15.46:0/3241049532 >> v2:172.21.15.121:3300/0 conn(0x7fb55815d2e0 0x7fb5580d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb556d9d640 1 -- 172.21.15.46:0/3241049532 >> v1:172.21.15.46:6789/0 conn(0x7fb55815d6c0 legacy=0x7fb55806b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb556d9d640 1 -- 172.21.15.46:0/3241049532 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55806c7b0 con 0x7fb55806bbd0 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb55559a640 1 -- 172.21.15.46:0/3241049532 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667439450 0 0) 0x7fb55806ce00 con 0x7fb55815d6c0 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.639+0000 7fb55659c640 1 --2- 172.21.15.46:0/3241049532 >> v2:172.21.15.121:3300/0 conn(0x7fb55815d2e0 0x7fb5580d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb556d9d640 1 --2- 172.21.15.46:0/3241049532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb5580d1b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb55808ccd0 tx=0x7fb55002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb44cc17d6571ad0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55559a640 1 -- 172.21.15.46:0/3241049532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550031870 con 0x7fb55806bbd0 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55559a640 1 -- 172.21.15.46:0/3241049532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5500319d0 con 0x7fb55806bbd0 2023-12-16T21:13:37.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55559a640 1 -- 172.21.15.46:0/3241049532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb550031d60 con 0x7fb55806bbd0 2023-12-16T21:13:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 -- 172.21.15.46:0/3241049532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 msgr2=0x7fb5580d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/3241049532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb5580d1b80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb55808ccd0 tx=0x7fb55002b910 comp rx=0 tx=0).stop 2023-12-16T21:13:37.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 -- 172.21.15.46:0/3241049532 shutdown_connections 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/3241049532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb5580d1b80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/3241049532 >> v2:172.21.15.121:3300/0 conn(0x7fb55815d2e0 0x7fb5580d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 -- 172.21.15.46:0/3241049532 >> 172.21.15.46:0/3241049532 conn(0x7fb55809c580 msgr2=0x7fb5580d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 -- 172.21.15.46:0/3241049532 shutdown_connections 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.640+0000 7fb55cf1d640 1 -- 172.21.15.46:0/3241049532 wait complete. 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 Processor -- start 2023-12-16T21:13:37.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 -- start start 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb55815a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55815d2e0 0x7fb55815ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55815d6c0 0x7fb558160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5581471d0 con 0x7fb55806bbd0 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb558159d70 con 0x7fb55815d2e0 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.642+0000 7fb55cf1d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c75c0 con 0x7fb55815d6c0 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb55815a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb55815a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54416/0 (socket says 172.21.15.46:54416) 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 -- 172.21.15.46:0/689059720 learned_addr learned my addr 172.21.15.46:0/689059720 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:37.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb556d9d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55815d6c0 0x7fb558160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 -- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55815d6c0 msgr2=0x7fb558160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55815d6c0 0x7fb558160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb555d9b640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55815d2e0 0x7fb55815ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 -- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55815d2e0 msgr2=0x7fb55815ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55815d2e0 0x7fb55815ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 -- 172.21.15.46:0/689059720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500094d0 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb555d9b640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55815d2e0 0x7fb55815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55659c640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb55815a200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb54800cf60 tx=0x7fb54800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54800d840 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54800d9a0 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54800dd30 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb558159740 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.643+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb558159a00 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.644+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb558067f40 con 0x7fb55806bbd0 2023-12-16T21:13:37.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.645+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb548015440 con 0x7fb55806bbd0 2023-12-16T21:13:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.645+0000 7fb54effd640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.645+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb548089f20 con 0x7fb55806bbd0 2023-12-16T21:13:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.645+0000 7fb555d9b640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:37.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.646+0000 7fb555d9b640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb55815daf0 tx=0x7fb540009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:37.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.649+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb548055110 con 0x7fb55806bbd0 2023-12-16T21:13:37.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:37.861+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb558067420 con 0x7fb55806bbd0 2023-12-16T21:13:38.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.524+0000 7fb54effd640 1 -- 172.21.15.46:0/689059720 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb54801a0c0 con 0x7fb55806bbd0 2023-12-16T21:13:38.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 msgr2=0x7fb53806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb55815daf0 tx=0x7fb540009040 comp rx=0 tx=0).stop 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 msgr2=0x7fb55815a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb55815a200 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb54800cf60 tx=0x7fb54800cf90 comp rx=0 tx=0).stop 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 shutdown_connections 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55815d6c0 0x7fb558160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55815d2e0 0x7fb55815ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 --2- 172.21.15.46:0/689059720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb55806bbd0 0x7fb55815a200 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 >> 172.21.15.46:0/689059720 conn(0x7fb55809c580 msgr2=0x7fb5580b1c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 shutdown_connections 2023-12-16T21:13:38.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.527+0000 7fb55cf1d640 1 -- 172.21.15.46:0/689059720 wait complete. 2023-12-16T21:13:38.540 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-16T21:13:38.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:13:38.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.653+0000 7f87a929b640 1 Processor -- start 2023-12-16T21:13:38.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.653+0000 7f87a929b640 1 -- start start 2023-12-16T21:13:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a929b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a40cb780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a929b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87a40b3400 0x7f87a40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a929b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87a40c7280 con 0x7f87a40ced00 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a929b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a40ce860 con 0x7f87a40b3400 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a929b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a40d09c0 con 0x7f87a4144b90 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87a40b3400 0x7f87a40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a27fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f87a40ced00 0x7f87a40b2cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33598/0 (socket says 172.21.15.46:33598) 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a27fc640 1 -- 172.21.15.46:0/3657845402 learned_addr learned my addr 172.21.15.46:0/3657845402 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.654+0000 7f87a2ffd640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a40cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a37fe640 1 -- 172.21.15.46:0/3657845402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 msgr2=0x7f87a40cb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a37fe640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a40cb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a37fe640 1 -- 172.21.15.46:0/3657845402 >> v1:172.21.15.46:6789/0 conn(0x7f87a40ced00 legacy=0x7f87a40b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a37fe640 1 -- 172.21.15.46:0/3657845402 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a40cbcc0 con 0x7f87a40b3400 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a1ffb640 1 -- 172.21.15.46:0/3657845402 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614759691 0 0) 0x7f87a40c7280 con 0x7f87a40ced00 2023-12-16T21:13:38.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a2ffd640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a40cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:38.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.655+0000 7f87a37fe640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40b3400 0x7f87a40d13d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8794009d90 tx=0x7f879402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f330c88842fe4a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a1ffb640 1 -- 172.21.15.46:0/3657845402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879402cc50 con 0x7f87a40b3400 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a1ffb640 1 -- 172.21.15.46:0/3657845402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879402cdb0 con 0x7f87a40b3400 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a1ffb640 1 -- 172.21.15.46:0/3657845402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8794034660 con 0x7f87a40b3400 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 -- 172.21.15.46:0/3657845402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40b3400 msgr2=0x7f87a40d13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40b3400 0x7f87a40d13d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8794009d90 tx=0x7f879402bea0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 -- 172.21.15.46:0/3657845402 shutdown_connections 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40b3400 0x7f87a40d13d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 --2- 172.21.15.46:0/3657845402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a40cb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 -- 172.21.15.46:0/3657845402 >> 172.21.15.46:0/3657845402 conn(0x7f87a406a4d0 msgr2=0x7f87a406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 -- 172.21.15.46:0/3657845402 shutdown_connections 2023-12-16T21:13:38.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.656+0000 7f87a929b640 1 -- 172.21.15.46:0/3657845402 wait complete. 2023-12-16T21:13:38.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 Processor -- start 2023-12-16T21:13:38.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 -- start start 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a40b3400 0x7f87a40ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 0x7f87a41e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a4143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41e4960 con 0x7f87a40ced00 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41e4c40 con 0x7f87a40b3400 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a929b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a41e4f20 con 0x7f87a4144b90 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 0x7f87a41e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a27fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 0x7f87a41e4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54424/0 (socket says 172.21.15.46:54424) 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.658+0000 7f87a27fc640 1 -- 172.21.15.46:0/2590235282 learned_addr learned my addr 172.21.15.46:0/2590235282 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a2ffd640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a40b3400 0x7f87a40ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a37fe640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a4143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a27fc640 1 -- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 msgr2=0x7f87a4143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a27fc640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a4143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a27fc640 1 -- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a40b3400 msgr2=0x7f87a40ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a27fc640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a40b3400 0x7f87a40ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a27fc640 1 -- 172.21.15.46:0/2590235282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8794009a70 con 0x7f87a40ced00 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a27fc640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 0x7f87a41e4130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f879800c4d0 tx=0x7f879800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980074e0 con 0x7f87a40ced00 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8798007660 con 0x7f87a40ced00 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a41e5200 con 0x7f87a40ced00 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798004a20 con 0x7f87a40ced00 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.659+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a41e4670 con 0x7f87a40ced00 2023-12-16T21:13:38.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.660+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a4067420 con 0x7f87a40ced00 2023-12-16T21:13:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.661+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8798004b80 con 0x7f87a40ced00 2023-12-16T21:13:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.661+0000 7f878b7fe640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f879006ae40 0x7f879006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.661+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8798088700 con 0x7f87a40ced00 2023-12-16T21:13:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.661+0000 7f87a2ffd640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f879006ae40 0x7f879006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:38.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.662+0000 7f87a2ffd640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f879006ae40 0x7f879006d300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f87a41f6580 tx=0x7f879c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:38.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.665+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8798053860 con 0x7f87a40ced00 2023-12-16T21:13:38.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.879+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87a40cbaa0 con 0x7f87a40ced00 2023-12-16T21:13:38.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.879+0000 7f878b7fe640 1 -- 172.21.15.46:0/2590235282 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f8798018090 con 0x7f87a40ced00 2023-12-16T21:13:38.879 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:38.879 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:13:38.880 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f879006ae40 msgr2=0x7f879006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f879006ae40 0x7f879006d300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f87a41f6580 tx=0x7f879c008040 comp rx=0 tx=0).stop 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 msgr2=0x7f87a41e4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 0x7f87a41e4130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f879800c4d0 tx=0x7f879800c990 comp rx=0 tx=0).stop 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 shutdown_connections 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f879006ae40 0x7f879006d300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a40ced00 0x7f87a41e4130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87a4144b90 0x7f87a4143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 --2- 172.21.15.46:0/2590235282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87a40b3400 0x7f87a40ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:38.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 >> 172.21.15.46:0/2590235282 conn(0x7f87a406a4d0 msgr2=0x7f87a40cc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:38.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 shutdown_connections 2023-12-16T21:13:38.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.881+0000 7f87a929b640 1 -- 172.21.15.46:0/2590235282 wait complete. 2023-12-16T21:13:38.892 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:38.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.998+0000 7faed5256640 1 Processor -- start 2023-12-16T21:13:38.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.998+0000 7faed5256640 1 -- start start 2023-12-16T21:13:38.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faed5256640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faed015d270 0x7faed006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faed5256640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed00709c0 0x7faed00cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:38.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faed5256640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed00d0c40 con 0x7faed00cdd10 2023-12-16T21:13:38.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faed5256640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed00ce1a0 con 0x7faed015d270 2023-12-16T21:13:38.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faed5256640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed00ced90 con 0x7faed00709c0 2023-12-16T21:13:38.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faeced9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faed015d270 0x7faed006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faeced9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7faed015d270 0x7faed006ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54432/0 (socket says 172.21.15.46:54432) 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faecf59e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7faed00cdd10 0x7faed00cc6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33600/0 (socket says 172.21.15.46:33600) 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faeced9d640 1 -- 172.21.15.46:0/519763873 learned_addr learned my addr 172.21.15.46:0/519763873 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:38.999+0000 7faece59c640 1 --2- 172.21.15.46:0/519763873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed00709c0 0x7faed00cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.000+0000 7faeced9d640 1 -- 172.21.15.46:0/519763873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed00709c0 msgr2=0x7faed00cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.000+0000 7faeced9d640 1 --2- 172.21.15.46:0/519763873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed00709c0 0x7faed00cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.000+0000 7faeced9d640 1 -- 172.21.15.46:0/519763873 >> v1:172.21.15.46:6789/0 conn(0x7faed00cdd10 legacy=0x7faed00cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.000+0000 7faeced9d640 1 -- 172.21.15.46:0/519763873 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed00cf070 con 0x7faed015d270 2023-12-16T21:13:39.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.000+0000 7faecdd9b640 1 -- 172.21.15.46:0/519763873 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187557096 0 0) 0x7faed00d0c40 con 0x7faed00cdd10 2023-12-16T21:13:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.000+0000 7faeced9d640 1 --2- 172.21.15.46:0/519763873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d270 0x7faed006ce70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faec0009870 tx=0x7faec002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2eb63f0cc556f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faecdd9b640 1 -- 172.21.15.46:0/519763873 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0031640 con 0x7faed015d270 2023-12-16T21:13:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faecdd9b640 1 -- 172.21.15.46:0/519763873 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec00317a0 con 0x7faed015d270 2023-12-16T21:13:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faecdd9b640 1 -- 172.21.15.46:0/519763873 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec0031b30 con 0x7faed015d270 2023-12-16T21:13:39.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 -- 172.21.15.46:0/519763873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d270 msgr2=0x7faed006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 --2- 172.21.15.46:0/519763873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d270 0x7faed006ce70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faec0009870 tx=0x7faec002b8f0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 -- 172.21.15.46:0/519763873 shutdown_connections 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 --2- 172.21.15.46:0/519763873 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed00709c0 0x7faed00cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 --2- 172.21.15.46:0/519763873 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed015d270 0x7faed006ce70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 -- 172.21.15.46:0/519763873 >> 172.21.15.46:0/519763873 conn(0x7faed00b2c10 msgr2=0x7faed00b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 -- 172.21.15.46:0/519763873 shutdown_connections 2023-12-16T21:13:39.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.001+0000 7faed5256640 1 -- 172.21.15.46:0/519763873 wait complete. 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 Processor -- start 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 -- start start 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 0x7faed01f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faed00cdd10 0x7faed01f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed015d270 0x7faed00c50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed015e6a0 con 0x7faed00709c0 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed015f1e0 con 0x7faed00cdd10 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faed5256640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed015e8b0 con 0x7faed015d270 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.003+0000 7faeced9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 0x7faed01f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 0x7faed01f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54438/0 (socket says 172.21.15.46:54438) 2023-12-16T21:13:39.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 -- 172.21.15.46:0/2458930678 learned_addr learned my addr 172.21.15.46:0/2458930678 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faecf59e640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed015d270 0x7faed00c50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faece59c640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faed00cdd10 0x7faed01f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 -- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed015d270 msgr2=0x7faed00c50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed015d270 0x7faed00c50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 -- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faed00cdd10 msgr2=0x7faed01f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faed00cdd10 0x7faed01f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 -- 172.21.15.46:0/2458930678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00094d0 con 0x7faed00709c0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeced9d640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 0x7faed01f6ce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faec0000c00 tx=0x7faec00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec003a930 con 0x7faed00709c0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec003aa90 con 0x7faed00709c0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed015eb90 con 0x7faed00709c0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec003ae40 con 0x7faed00709c0 2023-12-16T21:13:39.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.004+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed00c8ed0 con 0x7faed00709c0 2023-12-16T21:13:39.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.005+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed0067fc0 con 0x7faed00709c0 2023-12-16T21:13:39.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.006+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7faec002f050 con 0x7faed00709c0 2023-12-16T21:13:39.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.006+0000 7faeb77fe640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7faebc06ae40 0x7faebc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.006+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faec00ab2f0 con 0x7faed00709c0 2023-12-16T21:13:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.006+0000 7faece59c640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7faebc06ae40 0x7faebc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.007+0000 7faece59c640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7faebc06ae40 0x7faebc06d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faed00cb5d0 tx=0x7faec4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.010+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7faec0076450 con 0x7faed00709c0 2023-12-16T21:13:39.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.189+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faed0067420 con 0x7faed00709c0 2023-12-16T21:13:39.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.190+0000 7faeb77fe640 1 -- 172.21.15.46:0/2458930678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7faec0079a70 con 0x7faed00709c0 2023-12-16T21:13:39.190 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:39.190 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7faebc06ae40 msgr2=0x7faebc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7faebc06ae40 0x7faebc06d300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faed00cb5d0 tx=0x7faec4009190 comp rx=0 tx=0).stop 2023-12-16T21:13:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 msgr2=0x7faed01f6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 0x7faed01f6ce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faec0000c00 tx=0x7faec00046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 shutdown_connections 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7faebc06ae40 0x7faebc06d300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faed015d270 0x7faed00c50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faed00cdd10 0x7faed01f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 --2- 172.21.15.46:0/2458930678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed00709c0 0x7faed01f6ce0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.192+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 >> 172.21.15.46:0/2458930678 conn(0x7faed00b2c10 msgr2=0x7faed0073440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.193+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 shutdown_connections 2023-12-16T21:13:39.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.193+0000 7faed5256640 1 -- 172.21.15.46:0/2458930678 wait complete. 2023-12-16T21:13:39.203 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-16T21:13:39.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:39.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.309+0000 7fb696a51640 1 Processor -- start 2023-12-16T21:13:39.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.310+0000 7fb696a51640 1 -- start start 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.310+0000 7fb696a51640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6901448f0 0x7fb69015ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.310+0000 7fb696a51640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.310+0000 7fb696a51640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69015bf40 con 0x7fb69006a4d0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.310+0000 7fb696a51640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6900714c0 con 0x7fb6901448f0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.310+0000 7fb696a51640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69015f100 con 0x7fb69015b5d0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb69006a4d0 0x7fb690071bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33614/0 (socket says 172.21.15.46:33614) 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68f7fe640 1 -- 172.21.15.46:0/2234043526 learned_addr learned my addr 172.21.15.46:0/2234043526 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68ffff640 1 --2- 172.21.15.46:0/2234043526 >> v2:172.21.15.121:3300/0 conn(0x7fb6901448f0 0x7fb69015ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb694fee640 1 --2- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68effd640 1 -- 172.21.15.46:0/2234043526 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71518155 0 0) 0x7fb69015bf40 con 0x7fb69006a4d0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68effd640 1 -- 172.21.15.46:0/2234043526 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb67c003520 con 0x7fb69006a4d0 2023-12-16T21:13:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68ffff640 1 -- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 msgr2=0x7fb69015ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68ffff640 1 --2- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69015ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68ffff640 1 -- 172.21.15.46:0/2234043526 >> v1:172.21.15.46:6789/0 conn(0x7fb69006a4d0 legacy=0x7fb690071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68effd640 1 -- 172.21.15.46:0/2234043526 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1026797861 0 0) 0x7fb67c003520 con 0x7fb69006a4d0 2023-12-16T21:13:39.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.311+0000 7fb68ffff640 1 -- 172.21.15.46:0/2234043526 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69015f3a0 con 0x7fb6901448f0 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.312+0000 7fb68ffff640 1 --2- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb69015ebc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb6800097a0 tx=0x7fb68002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92090cb2d81fa615 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.312+0000 7fb68effd640 1 -- 172.21.15.46:0/2234043526 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678001690 con 0x7fb6901448f0 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.312+0000 7fb68effd640 1 -- 172.21.15.46:0/2234043526 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68002e050 con 0x7fb6901448f0 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.312+0000 7fb68effd640 1 -- 172.21.15.46:0/2234043526 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680030710 con 0x7fb6901448f0 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 -- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 msgr2=0x7fb69015ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 --2- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb69015ebc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb6800097a0 tx=0x7fb68002b790 comp rx=0 tx=0).stop 2023-12-16T21:13:39.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 -- 172.21.15.46:0/2234043526 shutdown_connections 2023-12-16T21:13:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 --2- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 --2- 172.21.15.46:0/2234043526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb69015ebc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 -- 172.21.15.46:0/2234043526 >> 172.21.15.46:0/2234043526 conn(0x7fb6900b1de0 msgr2=0x7fb6900b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 -- 172.21.15.46:0/2234043526 shutdown_connections 2023-12-16T21:13:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.313+0000 7fb696a51640 1 -- 172.21.15.46:0/2234043526 wait complete. 2023-12-16T21:13:39.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.314+0000 7fb696a51640 1 Processor -- start 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.314+0000 7fb696a51640 1 -- start start 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 0x7fb690138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb6901f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69013f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69013b3d0 con 0x7fb69006a4d0 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69013a7c0 con 0x7fb6901448f0 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69013aaa0 con 0x7fb69015b5d0 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 0x7fb690138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 0x7fb690138810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54440/0 (socket says 172.21.15.46:54440) 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 -- 172.21.15.46:0/2038605258 learned_addr learned my addr 172.21.15.46:0/2038605258 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68f7fe640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb6901f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb694fee640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69013f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 -- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 msgr2=0x7fb69013f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69013f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 -- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 msgr2=0x7fb6901f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb6901f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 -- 172.21.15.46:0/2038605258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680009400 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68ffff640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 0x7fb690138810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb680000c00 tx=0x7fb6800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68003c620 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68003c7a0 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69013ad80 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68003cb50 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.315+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb690138d50 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.316+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb690067420 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.317+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb680039070 con 0x7fb69006a4d0 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.317+0000 7fb68cff9640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb67006ae70 0x7fb67006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.317+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6800aab70 con 0x7fb69006a4d0 2023-12-16T21:13:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.317+0000 7fb68f7fe640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb67006ae70 0x7fb67006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.318+0000 7fb68f7fe640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb67006ae70 0x7fb67006d330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb678001eb0 tx=0x7fb678001c20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.321+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb680075af0 con 0x7fb69006a4d0 2023-12-16T21:13:39.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.492+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6900716d0 con 0x7fb67006ae70 2023-12-16T21:13:39.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.500+0000 7fb68cff9640 1 -- 172.21.15.46:0/2038605258 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+249803 (secure 0 0 0) 0x7fb67800e750 con 0x7fb67006ae70 2023-12-16T21:13:39.502 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:39.503 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:39.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb67006ae70 msgr2=0x7fb67006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb67006ae70 0x7fb67006d330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb678001eb0 tx=0x7fb678001c20 comp rx=0 tx=0).stop 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 msgr2=0x7fb690138810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 0x7fb690138810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb680000c00 tx=0x7fb6800046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 shutdown_connections 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb67006ae70 0x7fb67006d330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb69015b5d0 0x7fb69013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6901448f0 0x7fb6901f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 --2- 172.21.15.46:0/2038605258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb69006a4d0 0x7fb690138810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 >> 172.21.15.46:0/2038605258 conn(0x7fb6900b1de0 msgr2=0x7fb690142c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 shutdown_connections 2023-12-16T21:13:39.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.503+0000 7fb696a51640 1 -- 172.21.15.46:0/2038605258 wait complete. 2023-12-16T21:13:39.513 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-12-16T21:13:37.955182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":235400,"kb_used_data":2472,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739320,"statfs":{"total":773094113280,"available":772853063680,"internally_reserved":0,"allocated":2531328,"data_stored":1584048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000955"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-12-16T21:13:27.655698+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:27.655698+0000","last_peered":"2023-12-16T21:13:27.655698+0000","last_clean":"2023-12-16T21:13:27.655698+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:27.655698+0000","last_undegraded":"2023-12-16T21:13:27.655698+0000","last_fullsized":"2023-12-16T21:13:27.655698+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:39.518 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:39.554 INFO:tasks.ceph.mds.d.smithi121.stderr:2023-12-16T21:13:39.554+0000 7f29fa097640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:13:39.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.660+0000 7f9599768640 1 Processor -- start 2023-12-16T21:13:39.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.660+0000 7f9599768640 1 -- start start 2023-12-16T21:13:39.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f9599768640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415d2e0 0x7f95940d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f9599768640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959406ce00 con 0x7f959406bbd0 2023-12-16T21:13:39.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f9599768640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f959406c1f0 con 0x7f959415d6c0 2023-12-16T21:13:39.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f9599768640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959406c4d0 con 0x7f959415d2e0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f95937fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f959406bbd0 0x7f95940afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33616/0 (socket says 172.21.15.46:33616) 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f95937fe640 1 -- 172.21.15.46:0/1039772165 learned_addr learned my addr 172.21.15.46:0/1039772165 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.661+0000 7f9592ffd640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415d2e0 0x7f95940d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508004662 0 0) 0x7f959406ce00 con 0x7f959406bbd0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9580003520 con 0x7f959406bbd0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768506020 0 0) 0x7f959406c1f0 con 0x7f959415d6c0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f959406ce00 con 0x7f959415d6c0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3211204082 0 0) 0x7f9580003520 con 0x7f959406bbd0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415d2e0 msgr2=0x7f95940d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415d2e0 0x7f95940d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 >> v1:172.21.15.121:6789/0 conn(0x7f959415d6c0 legacy=0x7f959406b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959406c7b0 con 0x7f959406bbd0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f95840019f0 con 0x7f959406bbd0 2023-12-16T21:13:39.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 >> v1:172.21.15.46:6789/0 conn(0x7f959406bbd0 legacy=0x7f95940afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9580003b30 0x7f9580003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95800047c0 0x7f9580004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580005180 0x7f958000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959406c4d0 con 0x7f9580003b30 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958000c050 con 0x7f9580005180 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f958000c210 con 0x7f95800047c0 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415d2e0 0x7f95940d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9580003b30 0x7f9580003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f95927fc640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95800047c0 0x7f9580004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f95937fe640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580005180 0x7f958000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 -- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95800047c0 msgr2=0x7f9580004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95800047c0 0x7f9580004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 -- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580005180 msgr2=0x7f958000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:39.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580005180 0x7f958000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.662+0000 7f9592ffd640 1 -- 172.21.15.46:0/1039772165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9580003520 con 0x7f9580003b30 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.663+0000 7f9592ffd640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9580003b30 0x7f9580003f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95940b5d50 tx=0x7f95800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=750cb707575a92d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.663+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c004030 con 0x7f9580003b30 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.663+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958c004190 con 0x7f9580003b30 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.663+0000 7f9591ffb640 1 -- 172.21.15.46:0/1039772165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c004520 con 0x7f9580003b30 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f9599768640 1 -- 172.21.15.46:0/1039772165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9580003b30 msgr2=0x7f9580003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f9599768640 1 --2- 172.21.15.46:0/1039772165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9580003b30 0x7f9580003f30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f95940b5d50 tx=0x7f95800057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f95937fe640 1 -- 172.21.15.46:0/1039772165 reap_dead start 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f9599768640 1 -- 172.21.15.46:0/1039772165 shutdown_connections 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f9599768640 1 -- 172.21.15.46:0/1039772165 >> 172.21.15.46:0/1039772165 conn(0x7f959409c580 msgr2=0x7f95940d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f9599768640 1 -- 172.21.15.46:0/1039772165 shutdown_connections 2023-12-16T21:13:39.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.664+0000 7f9599768640 1 -- 172.21.15.46:0/1039772165 wait complete. 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.665+0000 7f9599768640 1 Processor -- start 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.665+0000 7f9599768640 1 -- start start 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9599768640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580003b30 0x7f95940b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9599768640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415feb0 0x7f9594160330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9599768640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 0x7f95940b8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9599768640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9594160870 con 0x7f95940b7bf0 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9599768640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95940b8ae0 con 0x7f9580003b30 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9599768640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95940ca440 con 0x7f959415feb0 2023-12-16T21:13:39.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 0x7f95940b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 0x7f95940b8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54446/0 (socket says 172.21.15.46:54446) 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 -- 172.21.15.46:0/500453046 learned_addr learned my addr 172.21.15.46:0/500453046 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95927fc640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415feb0 0x7f9594160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f9592ffd640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580003b30 0x7f95940b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 -- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415feb0 msgr2=0x7f9594160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415feb0 0x7f9594160330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 -- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580003b30 msgr2=0x7f95940b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580003b30 0x7f95940b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 -- 172.21.15.46:0/500453046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95940ca720 con 0x7f95940b7bf0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.666+0000 7f95937fe640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 0x7f95940b8080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f95940d2750 tx=0x7f958400a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.667+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958400ac00 con 0x7f95940b7bf0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.667+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958400ad80 con 0x7f95940b7bf0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.667+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95940caa00 con 0x7f95940b7bf0 2023-12-16T21:13:39.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.667+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584013630 con 0x7f95940b7bf0 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.667+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95940b85c0 con 0x7f95940b7bf0 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.668+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9594067420 con 0x7f95940b7bf0 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.668+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9584013790 con 0x7f95940b7bf0 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.669+0000 7f958b7fe640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f957406ae40 0x7f957406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.669+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9584089d70 con 0x7f95940b7bf0 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.669+0000 7f9592ffd640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f957406ae40 0x7f957406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:39.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.669+0000 7f9592ffd640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f957406ae40 0x7f957406d300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95940b5d50 tx=0x7f958c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:39.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.673+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9584054d80 con 0x7f95940b7bf0 2023-12-16T21:13:39.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.848+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95940ca2c0 con 0x7f957406ae40 2023-12-16T21:13:39.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.854+0000 7f958b7fe640 1 -- 172.21.15.46:0/500453046 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+249803 (secure 0 0 0) 0x7f958c0044d0 con 0x7f957406ae40 2023-12-16T21:13:39.856 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:39.858 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:39.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f957406ae40 msgr2=0x7f957406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f957406ae40 0x7f957406d300 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95940b5d50 tx=0x7f958c030000 comp rx=0 tx=0).stop 2023-12-16T21:13:39.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 msgr2=0x7f95940b8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 0x7f95940b8080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f95940d2750 tx=0x7f958400a600 comp rx=0 tx=0).stop 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 shutdown_connections 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f957406ae40 0x7f957406d300 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95940b7bf0 0x7f95940b8080 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f959415feb0 0x7f9594160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 --2- 172.21.15.46:0/500453046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9580003b30 0x7f95940b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 >> 172.21.15.46:0/500453046 conn(0x7f959409c580 msgr2=0x7f95940ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 shutdown_connections 2023-12-16T21:13:39.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:39.857+0000 7f9599768640 1 -- 172.21.15.46:0/500453046 wait complete. 2023-12-16T21:13:39.867 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-12-16T21:13:37.955182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":235400,"kb_used_data":2472,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739320,"statfs":{"total":773094113280,"available":772853063680,"internally_reserved":0,"allocated":2531328,"data_stored":1584048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000955"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-12-16T21:13:27.655698+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:27.655698+0000","last_peered":"2023-12-16T21:13:27.655698+0000","last_clean":"2023-12-16T21:13:27.655698+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:27.655698+0000","last_undegraded":"2023-12-16T21:13:27.655698+0000","last_fullsized":"2023-12-16T21:13:27.655698+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:35.487118+0000","last_change":"2023-12-16T21:13:35.487118+0000","last_active":"2023-12-16T21:13:35.487118+0000","last_peered":"2023-12-16T21:13:35.487118+0000","last_clean":"2023-12-16T21:13:35.487118+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:35.487118+0000","last_undegraded":"2023-12-16T21:13:35.487118+0000","last_fullsized":"2023-12-16T21:13:35.487118+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:13:34.483204+0000","last_change":"2023-12-16T21:13:34.483204+0000","last_active":"2023-12-16T21:13:34.483204+0000","last_peered":"2023-12-16T21:13:34.483204+0000","last_clean":"2023-12-16T21:13:34.483204+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:13:34.483204+0000","last_undegraded":"2023-12-16T21:13:34.483204+0000","last_fullsized":"2023-12-16T21:13:34.483204+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27760,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344080,"statfs":{"total":96636764160,"available":96608337920,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82299999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32048,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339792,"statfs":{"total":96636764160,"available":96603947008,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:39.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.015+0000 7f96a2456640 1 Processor -- start 2023-12-16T21:13:40.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.015+0000 7f96a2456640 1 -- start start 2023-12-16T21:13:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a2456640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f969c150560 0x7f969c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a2456640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f969c0b3390 0x7f969c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a2456640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150940 0x7f969c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a2456640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c0b49f0 con 0x7f969c0b3390 2023-12-16T21:13:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a2456640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969c0b3de0 con 0x7f969c150560 2023-12-16T21:13:40.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a2456640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0b40c0 con 0x7f969c150940 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f969b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f969c0b3390 0x7f969c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f969b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f969c0b3390 0x7f969c15d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54452/0 (socket says 172.21.15.46:54452) 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f969b7fe640 1 -- 172.21.15.46:0/1751186382 learned_addr learned my addr 172.21.15.46:0/1751186382 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f96a09f3640 1 --2- 172.21.15.46:0/1751186382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150940 0x7f969c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.016+0000 7f969bfff640 1 --2- 172.21.15.46:0/1751186382 >> v2:172.21.15.121:3300/0 conn(0x7f969c150560 0x7f969c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969b7fe640 1 -- 172.21.15.46:0/1751186382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150940 msgr2=0x7f969c15d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969b7fe640 1 --2- 172.21.15.46:0/1751186382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150940 0x7f969c15d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969b7fe640 1 -- 172.21.15.46:0/1751186382 >> v2:172.21.15.121:3300/0 conn(0x7f969c150560 msgr2=0x7f969c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969b7fe640 1 --2- 172.21.15.46:0/1751186382 >> v2:172.21.15.121:3300/0 conn(0x7f969c150560 0x7f969c0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969b7fe640 1 -- 172.21.15.46:0/1751186382 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0b43a0 con 0x7f969c0b3390 2023-12-16T21:13:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969b7fe640 1 --2- 172.21.15.46:0/1751186382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c15d200 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f969c072840 tx=0x7f968c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e3016b7f5bc3f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969affd640 1 -- 172.21.15.46:0/1751186382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c031640 con 0x7f969c0b3390 2023-12-16T21:13:40.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969affd640 1 -- 172.21.15.46:0/1751186382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c0317a0 con 0x7f969c0b3390 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.017+0000 7f969affd640 1 -- 172.21.15.46:0/1751186382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c031b30 con 0x7f969c0b3390 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 -- 172.21.15.46:0/1751186382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 msgr2=0x7f969c15d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 --2- 172.21.15.46:0/1751186382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c15d200 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f969c072840 tx=0x7f968c02b8f0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 -- 172.21.15.46:0/1751186382 shutdown_connections 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 --2- 172.21.15.46:0/1751186382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150940 0x7f969c15d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 --2- 172.21.15.46:0/1751186382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c15d200 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 --2- 172.21.15.46:0/1751186382 >> v2:172.21.15.121:3300/0 conn(0x7f969c150560 0x7f969c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 -- 172.21.15.46:0/1751186382 >> 172.21.15.46:0/1751186382 conn(0x7f969c06a4d0 msgr2=0x7f969c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 -- 172.21.15.46:0/1751186382 shutdown_connections 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.018+0000 7f96a2456640 1 -- 172.21.15.46:0/1751186382 wait complete. 2023-12-16T21:13:40.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.019+0000 7f96a2456640 1 Processor -- start 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.019+0000 7f96a2456640 1 -- start start 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a2456640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c1531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a2456640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150560 0x7f969c148760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a2456640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c158a30 0x7f969c15be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a2456640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c145950 con 0x7f969c0b3390 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a2456640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c146e30 con 0x7f969c158a30 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a2456640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c153730 con 0x7f969c150560 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f969bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c1531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f969bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c1531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54464/0 (socket says 172.21.15.46:54464) 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f969bfff640 1 -- 172.21.15.46:0/1849053448 learned_addr learned my addr 172.21.15.46:0/1849053448 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f969b7fe640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150560 0x7f969c148760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.020+0000 7f96a09f3640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c158a30 0x7f969c15be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f969bfff640 1 -- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150560 msgr2=0x7f969c148760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f969bfff640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150560 0x7f969c148760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f969bfff640 1 -- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c158a30 msgr2=0x7f969c15be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f969bfff640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c158a30 0x7f969c15be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f969bfff640 1 -- 172.21.15.46:0/1849053448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c0094d0 con 0x7f969c0b3390 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f969bfff640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c1531f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f969000d830 tx=0x7f969000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969000b6c0 con 0x7f969c0b3390 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969000b820 con 0x7f969c0b3390 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969000bbb0 con 0x7f969c0b3390 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c1597d0 con 0x7f969c0b3390 2023-12-16T21:13:40.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.021+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c148ca0 con 0x7f969c0b3390 2023-12-16T21:13:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.022+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c069400 con 0x7f969c0b3390 2023-12-16T21:13:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.022+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f969001e460 con 0x7f969c0b3390 2023-12-16T21:13:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.023+0000 7f9698ff9640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f968806ae40 0x7f968806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.023+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9690089bc0 con 0x7f969c0b3390 2023-12-16T21:13:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.023+0000 7f969b7fe640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f968806ae40 0x7f968806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.024+0000 7f969b7fe640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f968806ae40 0x7f968806d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f968c036000 tx=0x7f968c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.027+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9690054db0 con 0x7f969c0b3390 2023-12-16T21:13:40.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.200+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f969c0b4e80 con 0x7f968806ae40 2023-12-16T21:13:40.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.207+0000 7f9698ff9640 1 -- 172.21.15.46:0/1849053448 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264676 (secure 0 0 0) 0x7f968c044400 con 0x7f968806ae40 2023-12-16T21:13:40.210 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:40.211 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:40.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.210+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f968806ae40 msgr2=0x7f968806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.210+0000 7f96a2456640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f968806ae40 0x7f968806d300 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f968c036000 tx=0x7f968c03c040 comp rx=0 tx=0).stop 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 msgr2=0x7f969c1531f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c1531f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f969000d830 tx=0x7f969000dcf0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 shutdown_connections 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f968806ae40 0x7f968806d300 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f969c158a30 0x7f969c15be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f969c150560 0x7f969c148760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 --2- 172.21.15.46:0/1849053448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f969c0b3390 0x7f969c1531f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 >> 172.21.15.46:0/1849053448 conn(0x7f969c06a4d0 msgr2=0x7f969c15a300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 shutdown_connections 2023-12-16T21:13:40.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.211+0000 7f96a2456640 1 -- 172.21.15.46:0/1849053448 wait complete. 2023-12-16T21:13:40.221 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-16T21:13:39.955609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235664,"kb_used_data":2836,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739056,"statfs":{"total":773094113280,"available":772852793344,"internally_reserved":0,"allocated":2904064,"data_stored":1746104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.467658"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580448+0000","last_change":"2023-12-16T21:13:36.497094+0000","last_active":"2023-12-16T21:13:37.580448+0000","last_peered":"2023-12-16T21:13:37.580448+0000","last_clean":"2023-12-16T21:13:37.580448+0000","last_became_active":"2023-12-16T21:13:36.496839+0000","last_became_peered":"2023-12-16T21:13:36.496839+0000","last_unstale":"2023-12-16T21:13:37.580448+0000","last_undegraded":"2023-12-16T21:13:37.580448+0000","last_fullsized":"2023-12-16T21:13:37.580448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.499656+0000","last_change":"2023-12-16T21:13:36.496641+0000","last_active":"2023-12-16T21:13:36.499656+0000","last_peered":"2023-12-16T21:13:36.499656+0000","last_clean":"2023-12-16T21:13:36.499656+0000","last_became_active":"2023-12-16T21:13:36.496301+0000","last_became_peered":"2023-12-16T21:13:36.496301+0000","last_unstale":"2023-12-16T21:13:36.499656+0000","last_undegraded":"2023-12-16T21:13:36.499656+0000","last_fullsized":"2023-12-16T21:13:36.499656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:57:00.817822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.491825+0000","last_change":"2023-12-16T21:13:36.490436+0000","last_active":"2023-12-16T21:13:36.491825+0000","last_peered":"2023-12-16T21:13:36.491825+0000","last_clean":"2023-12-16T21:13:36.491825+0000","last_became_active":"2023-12-16T21:13:36.489881+0000","last_became_peered":"2023-12-16T21:13:36.489881+0000","last_unstale":"2023-12-16T21:13:36.491825+0000","last_undegraded":"2023-12-16T21:13:36.491825+0000","last_fullsized":"2023-12-16T21:13:36.491825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:06:58.471822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487045+0000","last_change":"2023-12-16T21:13:36.507326+0000","last_active":"2023-12-16T21:13:37.487045+0000","last_peered":"2023-12-16T21:13:37.487045+0000","last_clean":"2023-12-16T21:13:37.487045+0000","last_became_active":"2023-12-16T21:13:36.507063+0000","last_became_peered":"2023-12-16T21:13:36.507063+0000","last_unstale":"2023-12-16T21:13:37.487045+0000","last_undegraded":"2023-12-16T21:13:37.487045+0000","last_fullsized":"2023-12-16T21:13:37.487045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580048+0000","last_change":"2023-12-16T21:13:36.494596+0000","last_active":"2023-12-16T21:13:37.580048+0000","last_peered":"2023-12-16T21:13:37.580048+0000","last_clean":"2023-12-16T21:13:37.580048+0000","last_became_active":"2023-12-16T21:13:36.494238+0000","last_became_peered":"2023-12-16T21:13:36.494238+0000","last_unstale":"2023-12-16T21:13:37.580048+0000","last_undegraded":"2023-12-16T21:13:37.580048+0000","last_fullsized":"2023-12-16T21:13:37.580048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486248+0000","last_change":"2023-12-16T21:13:36.505505+0000","last_active":"2023-12-16T21:13:37.486248+0000","last_peered":"2023-12-16T21:13:37.486248+0000","last_clean":"2023-12-16T21:13:37.486248+0000","last_became_active":"2023-12-16T21:13:36.505205+0000","last_became_peered":"2023-12-16T21:13:36.505205+0000","last_unstale":"2023-12-16T21:13:37.486248+0000","last_undegraded":"2023-12-16T21:13:37.486248+0000","last_fullsized":"2023-12-16T21:13:37.486248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579992+0000","last_change":"2023-12-16T21:13:36.502009+0000","last_active":"2023-12-16T21:13:37.579992+0000","last_peered":"2023-12-16T21:13:37.579992+0000","last_clean":"2023-12-16T21:13:37.579992+0000","last_became_active":"2023-12-16T21:13:36.501672+0000","last_became_peered":"2023-12-16T21:13:36.501672+0000","last_unstale":"2023-12-16T21:13:37.579992+0000","last_undegraded":"2023-12-16T21:13:37.579992+0000","last_fullsized":"2023-12-16T21:13:37.579992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.484971+0000","last_change":"2023-12-16T21:13:36.502551+0000","last_active":"2023-12-16T21:13:37.484971+0000","last_peered":"2023-12-16T21:13:37.484971+0000","last_clean":"2023-12-16T21:13:37.484971+0000","last_became_active":"2023-12-16T21:13:36.502214+0000","last_became_peered":"2023-12-16T21:13:36.502214+0000","last_unstale":"2023-12-16T21:13:37.484971+0000","last_undegraded":"2023-12-16T21:13:37.484971+0000","last_fullsized":"2023-12-16T21:13:37.484971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580474+0000","last_change":"2023-12-16T21:13:36.507546+0000","last_active":"2023-12-16T21:13:37.580474+0000","last_peered":"2023-12-16T21:13:37.580474+0000","last_clean":"2023-12-16T21:13:37.580474+0000","last_became_active":"2023-12-16T21:13:36.507253+0000","last_became_peered":"2023-12-16T21:13:36.507253+0000","last_unstale":"2023-12-16T21:13:37.580474+0000","last_undegraded":"2023-12-16T21:13:37.580474+0000","last_fullsized":"2023-12-16T21:13:37.580474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:22:24.104289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485901+0000","last_change":"2023-12-16T21:13:36.508602+0000","last_active":"2023-12-16T21:13:37.485901+0000","last_peered":"2023-12-16T21:13:37.485901+0000","last_clean":"2023-12-16T21:13:37.485901+0000","last_became_active":"2023-12-16T21:13:36.508223+0000","last_became_peered":"2023-12-16T21:13:36.508223+0000","last_unstale":"2023-12-16T21:13:37.485901+0000","last_undegraded":"2023-12-16T21:13:37.485901+0000","last_fullsized":"2023-12-16T21:13:37.485901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485806+0000","last_change":"2023-12-16T21:13:36.508624+0000","last_active":"2023-12-16T21:13:37.485806+0000","last_peered":"2023-12-16T21:13:37.485806+0000","last_clean":"2023-12-16T21:13:37.485806+0000","last_became_active":"2023-12-16T21:13:36.508293+0000","last_became_peered":"2023-12-16T21:13:36.508293+0000","last_unstale":"2023-12-16T21:13:37.485806+0000","last_undegraded":"2023-12-16T21:13:37.485806+0000","last_fullsized":"2023-12-16T21:13:37.485806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.509037+0000","last_change":"2023-12-16T21:13:36.499195+0000","last_active":"2023-12-16T21:13:36.509037+0000","last_peered":"2023-12-16T21:13:36.509037+0000","last_clean":"2023-12-16T21:13:36.509037+0000","last_became_active":"2023-12-16T21:13:36.498927+0000","last_became_peered":"2023-12-16T21:13:36.498927+0000","last_unstale":"2023-12-16T21:13:36.509037+0000","last_undegraded":"2023-12-16T21:13:36.509037+0000","last_fullsized":"2023-12-16T21:13:36.509037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:31:34.903149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485287+0000","last_change":"2023-12-16T21:13:36.494278+0000","last_active":"2023-12-16T21:13:37.485287+0000","last_peered":"2023-12-16T21:13:37.485287+0000","last_clean":"2023-12-16T21:13:37.485287+0000","last_became_active":"2023-12-16T21:13:36.493970+0000","last_became_peered":"2023-12-16T21:13:36.493970+0000","last_unstale":"2023-12-16T21:13:37.485287+0000","last_undegraded":"2023-12-16T21:13:37.485287+0000","last_fullsized":"2023-12-16T21:13:37.485287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501464+0000","last_change":"2023-12-16T21:13:36.499374+0000","last_active":"2023-12-16T21:13:36.501464+0000","last_peered":"2023-12-16T21:13:36.501464+0000","last_clean":"2023-12-16T21:13:36.501464+0000","last_became_active":"2023-12-16T21:13:36.499097+0000","last_became_peered":"2023-12-16T21:13:36.499097+0000","last_unstale":"2023-12-16T21:13:36.501464+0000","last_undegraded":"2023-12-16T21:13:36.501464+0000","last_fullsized":"2023-12-16T21:13:36.501464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:53.131591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.505430+0000","last_change":"2023-12-16T21:13:36.504350+0000","last_active":"2023-12-16T21:13:36.505430+0000","last_peered":"2023-12-16T21:13:36.505430+0000","last_clean":"2023-12-16T21:13:36.505430+0000","last_became_active":"2023-12-16T21:13:36.504046+0000","last_became_peered":"2023-12-16T21:13:36.504046+0000","last_unstale":"2023-12-16T21:13:36.505430+0000","last_undegraded":"2023-12-16T21:13:36.505430+0000","last_fullsized":"2023-12-16T21:13:36.505430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485909+0000","last_change":"2023-12-16T21:13:36.502440+0000","last_active":"2023-12-16T21:13:37.485909+0000","last_peered":"2023-12-16T21:13:37.485909+0000","last_clean":"2023-12-16T21:13:37.485909+0000","last_became_active":"2023-12-16T21:13:36.502160+0000","last_became_peered":"2023-12-16T21:13:36.502160+0000","last_unstale":"2023-12-16T21:13:37.485909+0000","last_undegraded":"2023-12-16T21:13:37.485909+0000","last_fullsized":"2023-12-16T21:13:37.485909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487000+0000","last_change":"2023-12-16T21:13:36.508778+0000","last_active":"2023-12-16T21:13:37.487000+0000","last_peered":"2023-12-16T21:13:37.487000+0000","last_clean":"2023-12-16T21:13:37.487000+0000","last_became_active":"2023-12-16T21:13:36.508506+0000","last_became_peered":"2023-12-16T21:13:36.508506+0000","last_unstale":"2023-12-16T21:13:37.487000+0000","last_undegraded":"2023-12-16T21:13:37.487000+0000","last_fullsized":"2023-12-16T21:13:37.487000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.495755+0000","last_change":"2023-12-16T21:13:36.494459+0000","last_active":"2023-12-16T21:13:36.495755+0000","last_peered":"2023-12-16T21:13:36.495755+0000","last_clean":"2023-12-16T21:13:36.495755+0000","last_became_active":"2023-12-16T21:13:36.494156+0000","last_became_peered":"2023-12-16T21:13:36.494156+0000","last_unstale":"2023-12-16T21:13:36.495755+0000","last_undegraded":"2023-12-16T21:13:36.495755+0000","last_fullsized":"2023-12-16T21:13:36.495755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:47.663889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580395+0000","last_change":"2023-12-16T21:13:36.496446+0000","last_active":"2023-12-16T21:13:37.580395+0000","last_peered":"2023-12-16T21:13:37.580395+0000","last_clean":"2023-12-16T21:13:37.580395+0000","last_became_active":"2023-12-16T21:13:36.496123+0000","last_became_peered":"2023-12-16T21:13:36.496123+0000","last_unstale":"2023-12-16T21:13:37.580395+0000","last_undegraded":"2023-12-16T21:13:37.580395+0000","last_fullsized":"2023-12-16T21:13:37.580395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486172+0000","last_change":"2023-12-16T21:13:36.505223+0000","last_active":"2023-12-16T21:13:37.486172+0000","last_peered":"2023-12-16T21:13:37.486172+0000","last_clean":"2023-12-16T21:13:37.486172+0000","last_became_active":"2023-12-16T21:13:36.504834+0000","last_became_peered":"2023-12-16T21:13:36.504834+0000","last_unstale":"2023-12-16T21:13:37.486172+0000","last_undegraded":"2023-12-16T21:13:37.486172+0000","last_fullsized":"2023-12-16T21:13:37.486172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501391+0000","last_change":"2023-12-16T21:13:36.500880+0000","last_active":"2023-12-16T21:13:36.501391+0000","last_peered":"2023-12-16T21:13:36.501391+0000","last_clean":"2023-12-16T21:13:36.501391+0000","last_became_active":"2023-12-16T21:13:36.500532+0000","last_became_peered":"2023-12-16T21:13:36.500532+0000","last_unstale":"2023-12-16T21:13:36.501391+0000","last_undegraded":"2023-12-16T21:13:36.501391+0000","last_fullsized":"2023-12-16T21:13:36.501391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486491+0000","last_change":"2023-12-16T21:13:36.507578+0000","last_active":"2023-12-16T21:13:37.486491+0000","last_peered":"2023-12-16T21:13:37.486491+0000","last_clean":"2023-12-16T21:13:37.486491+0000","last_became_active":"2023-12-16T21:13:36.507206+0000","last_became_peered":"2023-12-16T21:13:36.507206+0000","last_unstale":"2023-12-16T21:13:37.486491+0000","last_undegraded":"2023-12-16T21:13:37.486491+0000","last_fullsized":"2023-12-16T21:13:37.486491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485167+0000","last_change":"2023-12-16T21:13:36.506029+0000","last_active":"2023-12-16T21:13:37.485167+0000","last_peered":"2023-12-16T21:13:37.485167+0000","last_clean":"2023-12-16T21:13:37.485167+0000","last_became_active":"2023-12-16T21:13:36.505785+0000","last_became_peered":"2023-12-16T21:13:36.505785+0000","last_unstale":"2023-12-16T21:13:37.485167+0000","last_undegraded":"2023-12-16T21:13:37.485167+0000","last_fullsized":"2023-12-16T21:13:37.485167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486054+0000","last_change":"2023-12-16T21:13:36.504916+0000","last_active":"2023-12-16T21:13:37.486054+0000","last_peered":"2023-12-16T21:13:37.486054+0000","last_clean":"2023-12-16T21:13:37.486054+0000","last_became_active":"2023-12-16T21:13:36.504422+0000","last_became_peered":"2023-12-16T21:13:36.504422+0000","last_unstale":"2023-12-16T21:13:37.486054+0000","last_undegraded":"2023-12-16T21:13:37.486054+0000","last_fullsized":"2023-12-16T21:13:37.486054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.503543+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.503210+0000","last_became_peered":"2023-12-16T21:13:36.503210+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486065+0000","last_change":"2023-12-16T21:13:36.494611+0000","last_active":"2023-12-16T21:13:37.486065+0000","last_peered":"2023-12-16T21:13:37.486065+0000","last_clean":"2023-12-16T21:13:37.486065+0000","last_became_active":"2023-12-16T21:13:36.494373+0000","last_became_peered":"2023-12-16T21:13:36.494373+0000","last_unstale":"2023-12-16T21:13:37.486065+0000","last_undegraded":"2023-12-16T21:13:37.486065+0000","last_fullsized":"2023-12-16T21:13:37.486065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.500812+0000","last_change":"2023-12-16T21:13:36.498640+0000","last_active":"2023-12-16T21:13:36.500812+0000","last_peered":"2023-12-16T21:13:36.500812+0000","last_clean":"2023-12-16T21:13:36.500812+0000","last_became_active":"2023-12-16T21:13:36.498334+0000","last_became_peered":"2023-12-16T21:13:36.498334+0000","last_unstale":"2023-12-16T21:13:36.500812+0000","last_undegraded":"2023-12-16T21:13:36.500812+0000","last_fullsized":"2023-12-16T21:13:36.500812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485419+0000","last_change":"2023-12-16T21:13:36.506001+0000","last_active":"2023-12-16T21:13:37.485419+0000","last_peered":"2023-12-16T21:13:37.485419+0000","last_clean":"2023-12-16T21:13:37.485419+0000","last_became_active":"2023-12-16T21:13:36.505724+0000","last_became_peered":"2023-12-16T21:13:36.505724+0000","last_unstale":"2023-12-16T21:13:37.485419+0000","last_undegraded":"2023-12-16T21:13:37.485419+0000","last_fullsized":"2023-12-16T21:13:37.485419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.491115+0000","last_change":"2023-12-16T21:13:36.489974+0000","last_active":"2023-12-16T21:13:36.491115+0000","last_peered":"2023-12-16T21:13:36.491115+0000","last_clean":"2023-12-16T21:13:36.491115+0000","last_became_active":"2023-12-16T21:13:36.489597+0000","last_became_peered":"2023-12-16T21:13:36.489597+0000","last_unstale":"2023-12-16T21:13:36.491115+0000","last_undegraded":"2023-12-16T21:13:36.491115+0000","last_fullsized":"2023-12-16T21:13:36.491115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:46.012193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486570+0000","last_change":"2023-12-16T21:13:36.505535+0000","last_active":"2023-12-16T21:13:37.486570+0000","last_peered":"2023-12-16T21:13:37.486570+0000","last_clean":"2023-12-16T21:13:37.486570+0000","last_became_active":"2023-12-16T21:13:36.505185+0000","last_became_peered":"2023-12-16T21:13:36.505185+0000","last_unstale":"2023-12-16T21:13:37.486570+0000","last_undegraded":"2023-12-16T21:13:37.486570+0000","last_fullsized":"2023-12-16T21:13:37.486570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.505879+0000","last_change":"2023-12-16T21:13:36.504923+0000","last_active":"2023-12-16T21:13:36.505879+0000","last_peered":"2023-12-16T21:13:36.505879+0000","last_clean":"2023-12-16T21:13:36.505879+0000","last_became_active":"2023-12-16T21:13:36.504451+0000","last_became_peered":"2023-12-16T21:13:36.504451+0000","last_unstale":"2023-12-16T21:13:36.505879+0000","last_undegraded":"2023-12-16T21:13:36.505879+0000","last_fullsized":"2023-12-16T21:13:36.505879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:06:58.471822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485417+0000","last_change":"2023-12-16T21:13:36.497205+0000","last_active":"2023-12-16T21:13:37.485417+0000","last_peered":"2023-12-16T21:13:37.485417+0000","last_clean":"2023-12-16T21:13:37.485417+0000","last_became_active":"2023-12-16T21:13:36.496919+0000","last_became_peered":"2023-12-16T21:13:36.496919+0000","last_unstale":"2023-12-16T21:13:37.485417+0000","last_undegraded":"2023-12-16T21:13:37.485417+0000","last_fullsized":"2023-12-16T21:13:37.485417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.511775+0000","last_change":"2023-12-16T21:13:36.503295+0000","last_active":"2023-12-16T21:13:36.511775+0000","last_peered":"2023-12-16T21:13:36.511775+0000","last_clean":"2023-12-16T21:13:36.511775+0000","last_became_active":"2023-12-16T21:13:36.502730+0000","last_became_peered":"2023-12-16T21:13:36.502730+0000","last_unstale":"2023-12-16T21:13:36.511775+0000","last_undegraded":"2023-12-16T21:13:36.511775+0000","last_fullsized":"2023-12-16T21:13:36.511775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:06:58.471822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485372+0000","last_change":"2023-12-16T21:13:35.508997+0000","last_active":"2023-12-16T21:13:37.485372+0000","last_peered":"2023-12-16T21:13:37.485372+0000","last_clean":"2023-12-16T21:13:37.485372+0000","last_became_active":"2023-12-16T21:13:35.508689+0000","last_became_peered":"2023-12-16T21:13:35.508689+0000","last_unstale":"2023-12-16T21:13:37.485372+0000","last_undegraded":"2023-12-16T21:13:37.485372+0000","last_fullsized":"2023-12-16T21:13:37.485372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.504701+0000","last_change":"2023-12-16T21:13:36.504100+0000","last_active":"2023-12-16T21:13:36.504701+0000","last_peered":"2023-12-16T21:13:36.504701+0000","last_clean":"2023-12-16T21:13:36.504701+0000","last_became_active":"2023-12-16T21:13:36.503340+0000","last_became_peered":"2023-12-16T21:13:36.503340+0000","last_unstale":"2023-12-16T21:13:36.504701+0000","last_undegraded":"2023-12-16T21:13:36.504701+0000","last_fullsized":"2023-12-16T21:13:36.504701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485542+0000","last_change":"2023-12-16T21:13:35.508135+0000","last_active":"2023-12-16T21:13:37.485542+0000","last_peered":"2023-12-16T21:13:37.485542+0000","last_clean":"2023-12-16T21:13:37.485542+0000","last_became_active":"2023-12-16T21:13:35.507149+0000","last_became_peered":"2023-12-16T21:13:35.507149+0000","last_unstale":"2023-12-16T21:13:37.485542+0000","last_undegraded":"2023-12-16T21:13:37.485542+0000","last_fullsized":"2023-12-16T21:13:37.485542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485144+0000","last_change":"2023-12-16T21:13:35.510923+0000","last_active":"2023-12-16T21:13:37.485144+0000","last_peered":"2023-12-16T21:13:37.485144+0000","last_clean":"2023-12-16T21:13:37.485144+0000","last_became_active":"2023-12-16T21:13:35.510638+0000","last_became_peered":"2023-12-16T21:13:35.510638+0000","last_unstale":"2023-12-16T21:13:37.485144+0000","last_undegraded":"2023-12-16T21:13:37.485144+0000","last_fullsized":"2023-12-16T21:13:37.485144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:52.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485039+0000","last_change":"2023-12-16T21:13:36.490652+0000","last_active":"2023-12-16T21:13:37.485039+0000","last_peered":"2023-12-16T21:13:37.485039+0000","last_clean":"2023-12-16T21:13:37.485039+0000","last_became_active":"2023-12-16T21:13:36.489990+0000","last_became_peered":"2023-12-16T21:13:36.489990+0000","last_unstale":"2023-12-16T21:13:37.485039+0000","last_undegraded":"2023-12-16T21:13:37.485039+0000","last_fullsized":"2023-12-16T21:13:37.485039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486149+0000","last_change":"2023-12-16T21:13:35.498392+0000","last_active":"2023-12-16T21:13:37.486149+0000","last_peered":"2023-12-16T21:13:37.486149+0000","last_clean":"2023-12-16T21:13:37.486149+0000","last_became_active":"2023-12-16T21:13:35.498119+0000","last_became_peered":"2023-12-16T21:13:35.498119+0000","last_unstale":"2023-12-16T21:13:37.486149+0000","last_undegraded":"2023-12-16T21:13:37.486149+0000","last_fullsized":"2023-12-16T21:13:37.486149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.500917+0000","last_change":"2023-12-16T21:13:36.499009+0000","last_active":"2023-12-16T21:13:36.500917+0000","last_peered":"2023-12-16T21:13:36.500917+0000","last_clean":"2023-12-16T21:13:36.500917+0000","last_became_active":"2023-12-16T21:13:36.498714+0000","last_became_peered":"2023-12-16T21:13:36.498714+0000","last_unstale":"2023-12-16T21:13:36.500917+0000","last_undegraded":"2023-12-16T21:13:36.500917+0000","last_fullsized":"2023-12-16T21:13:36.500917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:26:23.745820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485912+0000","last_change":"2023-12-16T21:13:35.508895+0000","last_active":"2023-12-16T21:13:37.485912+0000","last_peered":"2023-12-16T21:13:37.485912+0000","last_clean":"2023-12-16T21:13:37.485912+0000","last_became_active":"2023-12-16T21:13:35.508644+0000","last_became_peered":"2023-12-16T21:13:35.508644+0000","last_unstale":"2023-12-16T21:13:37.485912+0000","last_undegraded":"2023-12-16T21:13:37.485912+0000","last_fullsized":"2023-12-16T21:13:37.485912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486627+0000","last_change":"2023-12-16T21:13:36.508270+0000","last_active":"2023-12-16T21:13:37.486627+0000","last_peered":"2023-12-16T21:13:37.486627+0000","last_clean":"2023-12-16T21:13:37.486627+0000","last_became_active":"2023-12-16T21:13:36.507998+0000","last_became_peered":"2023-12-16T21:13:36.507998+0000","last_unstale":"2023-12-16T21:13:37.486627+0000","last_undegraded":"2023-12-16T21:13:37.486627+0000","last_fullsized":"2023-12-16T21:13:37.486627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486639+0000","last_change":"2023-12-16T21:13:35.508588+0000","last_active":"2023-12-16T21:13:37.486639+0000","last_peered":"2023-12-16T21:13:37.486639+0000","last_clean":"2023-12-16T21:13:37.486639+0000","last_became_active":"2023-12-16T21:13:35.508246+0000","last_became_peered":"2023-12-16T21:13:35.508246+0000","last_unstale":"2023-12-16T21:13:37.486639+0000","last_undegraded":"2023-12-16T21:13:37.486639+0000","last_fullsized":"2023-12-16T21:13:37.486639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.508173+0000","last_change":"2023-12-16T21:13:36.507570+0000","last_active":"2023-12-16T21:13:36.508173+0000","last_peered":"2023-12-16T21:13:36.508173+0000","last_clean":"2023-12-16T21:13:36.508173+0000","last_became_active":"2023-12-16T21:13:36.507226+0000","last_became_peered":"2023-12-16T21:13:36.507226+0000","last_unstale":"2023-12-16T21:13:36.508173+0000","last_undegraded":"2023-12-16T21:13:36.508173+0000","last_fullsized":"2023-12-16T21:13:36.508173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.526963+0000","last_change":"2023-12-16T21:13:35.509646+0000","last_active":"2023-12-16T21:13:36.526963+0000","last_peered":"2023-12-16T21:13:36.526963+0000","last_clean":"2023-12-16T21:13:36.526963+0000","last_became_active":"2023-12-16T21:13:35.509335+0000","last_became_peered":"2023-12-16T21:13:35.509335+0000","last_unstale":"2023-12-16T21:13:36.526963+0000","last_undegraded":"2023-12-16T21:13:36.526963+0000","last_fullsized":"2023-12-16T21:13:36.526963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.509063+0000","last_change":"2023-12-16T21:13:36.499896+0000","last_active":"2023-12-16T21:13:36.509063+0000","last_peered":"2023-12-16T21:13:36.509063+0000","last_clean":"2023-12-16T21:13:36.509063+0000","last_became_active":"2023-12-16T21:13:36.499625+0000","last_became_peered":"2023-12-16T21:13:36.499625+0000","last_unstale":"2023-12-16T21:13:36.509063+0000","last_undegraded":"2023-12-16T21:13:36.509063+0000","last_fullsized":"2023-12-16T21:13:36.509063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:10:06.023454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486051+0000","last_change":"2023-12-16T21:13:35.507588+0000","last_active":"2023-12-16T21:13:37.486051+0000","last_peered":"2023-12-16T21:13:37.486051+0000","last_clean":"2023-12-16T21:13:37.486051+0000","last_became_active":"2023-12-16T21:13:35.507264+0000","last_became_peered":"2023-12-16T21:13:35.507264+0000","last_unstale":"2023-12-16T21:13:37.486051+0000","last_undegraded":"2023-12-16T21:13:37.486051+0000","last_fullsized":"2023-12-16T21:13:37.486051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580181+0000","last_change":"2023-12-16T21:13:36.495697+0000","last_active":"2023-12-16T21:13:37.580181+0000","last_peered":"2023-12-16T21:13:37.580181+0000","last_clean":"2023-12-16T21:13:37.580181+0000","last_became_active":"2023-12-16T21:13:36.495099+0000","last_became_peered":"2023-12-16T21:13:36.495099+0000","last_unstale":"2023-12-16T21:13:37.580181+0000","last_undegraded":"2023-12-16T21:13:37.580181+0000","last_fullsized":"2023-12-16T21:13:37.580181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.529125+0000","last_change":"2023-12-16T21:13:35.502116+0000","last_active":"2023-12-16T21:13:36.529125+0000","last_peered":"2023-12-16T21:13:36.529125+0000","last_clean":"2023-12-16T21:13:36.529125+0000","last_became_active":"2023-12-16T21:13:35.501814+0000","last_became_peered":"2023-12-16T21:13:35.501814+0000","last_unstale":"2023-12-16T21:13:36.529125+0000","last_undegraded":"2023-12-16T21:13:36.529125+0000","last_fullsized":"2023-12-16T21:13:36.529125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501440+0000","last_change":"2023-12-16T21:13:36.499212+0000","last_active":"2023-12-16T21:13:36.501440+0000","last_peered":"2023-12-16T21:13:36.501440+0000","last_clean":"2023-12-16T21:13:36.501440+0000","last_became_active":"2023-12-16T21:13:36.498745+0000","last_became_peered":"2023-12-16T21:13:36.498745+0000","last_unstale":"2023-12-16T21:13:36.501440+0000","last_undegraded":"2023-12-16T21:13:36.501440+0000","last_fullsized":"2023-12-16T21:13:36.501440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:26.560123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486589+0000","last_change":"2023-12-16T21:13:35.506529+0000","last_active":"2023-12-16T21:13:37.486589+0000","last_peered":"2023-12-16T21:13:37.486589+0000","last_clean":"2023-12-16T21:13:37.486589+0000","last_became_active":"2023-12-16T21:13:35.506231+0000","last_became_peered":"2023-12-16T21:13:35.506231+0000","last_unstale":"2023-12-16T21:13:37.486589+0000","last_undegraded":"2023-12-16T21:13:37.486589+0000","last_fullsized":"2023-12-16T21:13:37.486589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580108+0000","last_change":"2023-12-16T21:13:36.504072+0000","last_active":"2023-12-16T21:13:37.580108+0000","last_peered":"2023-12-16T21:13:37.580108+0000","last_clean":"2023-12-16T21:13:37.580108+0000","last_became_active":"2023-12-16T21:13:36.503801+0000","last_became_peered":"2023-12-16T21:13:36.503801+0000","last_unstale":"2023-12-16T21:13:37.580108+0000","last_undegraded":"2023-12-16T21:13:37.580108+0000","last_fullsized":"2023-12-16T21:13:37.580108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485664+0000","last_change":"2023-12-16T21:13:35.508204+0000","last_active":"2023-12-16T21:13:37.485664+0000","last_peered":"2023-12-16T21:13:37.485664+0000","last_clean":"2023-12-16T21:13:37.485664+0000","last_became_active":"2023-12-16T21:13:35.507929+0000","last_became_peered":"2023-12-16T21:13:35.507929+0000","last_unstale":"2023-12-16T21:13:37.485664+0000","last_undegraded":"2023-12-16T21:13:37.485664+0000","last_fullsized":"2023-12-16T21:13:37.485664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485428+0000","last_change":"2023-12-16T21:13:36.497155+0000","last_active":"2023-12-16T21:13:37.485428+0000","last_peered":"2023-12-16T21:13:37.485428+0000","last_clean":"2023-12-16T21:13:37.485428+0000","last_became_active":"2023-12-16T21:13:36.496862+0000","last_became_peered":"2023-12-16T21:13:36.496862+0000","last_unstale":"2023-12-16T21:13:37.485428+0000","last_undegraded":"2023-12-16T21:13:37.485428+0000","last_fullsized":"2023-12-16T21:13:37.485428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.525464+0000","last_change":"2023-12-16T21:13:35.508118+0000","last_active":"2023-12-16T21:13:36.525464+0000","last_peered":"2023-12-16T21:13:36.525464+0000","last_clean":"2023-12-16T21:13:36.525464+0000","last_became_active":"2023-12-16T21:13:35.507827+0000","last_became_peered":"2023-12-16T21:13:35.507827+0000","last_unstale":"2023-12-16T21:13:36.525464+0000","last_undegraded":"2023-12-16T21:13:36.525464+0000","last_fullsized":"2023-12-16T21:13:36.525464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580055+0000","last_change":"2023-12-16T21:13:36.504720+0000","last_active":"2023-12-16T21:13:37.580055+0000","last_peered":"2023-12-16T21:13:37.580055+0000","last_clean":"2023-12-16T21:13:37.580055+0000","last_became_active":"2023-12-16T21:13:36.504444+0000","last_became_peered":"2023-12-16T21:13:36.504444+0000","last_unstale":"2023-12-16T21:13:37.580055+0000","last_undegraded":"2023-12-16T21:13:37.580055+0000","last_fullsized":"2023-12-16T21:13:37.580055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.524253+0000","last_change":"2023-12-16T21:13:35.509590+0000","last_active":"2023-12-16T21:13:36.524253+0000","last_peered":"2023-12-16T21:13:36.524253+0000","last_clean":"2023-12-16T21:13:36.524253+0000","last_became_active":"2023-12-16T21:13:35.509278+0000","last_became_peered":"2023-12-16T21:13:35.509278+0000","last_unstale":"2023-12-16T21:13:36.524253+0000","last_undegraded":"2023-12-16T21:13:36.524253+0000","last_fullsized":"2023-12-16T21:13:36.524253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:46.659974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580433+0000","last_change":"2023-12-16T21:13:36.498151+0000","last_active":"2023-12-16T21:13:37.580433+0000","last_peered":"2023-12-16T21:13:37.580433+0000","last_clean":"2023-12-16T21:13:37.580433+0000","last_became_active":"2023-12-16T21:13:36.497907+0000","last_became_peered":"2023-12-16T21:13:36.497907+0000","last_unstale":"2023-12-16T21:13:37.580433+0000","last_undegraded":"2023-12-16T21:13:37.580433+0000","last_fullsized":"2023-12-16T21:13:37.580433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.523929+0000","last_change":"2023-12-16T21:13:35.507707+0000","last_active":"2023-12-16T21:13:36.523929+0000","last_peered":"2023-12-16T21:13:36.523929+0000","last_clean":"2023-12-16T21:13:36.523929+0000","last_became_active":"2023-12-16T21:13:35.507148+0000","last_became_peered":"2023-12-16T21:13:35.507148+0000","last_unstale":"2023-12-16T21:13:36.523929+0000","last_undegraded":"2023-12-16T21:13:36.523929+0000","last_fullsized":"2023-12-16T21:13:36.523929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580539+0000","last_change":"2023-12-16T21:13:36.503790+0000","last_active":"2023-12-16T21:13:37.580539+0000","last_peered":"2023-12-16T21:13:37.580539+0000","last_clean":"2023-12-16T21:13:37.580539+0000","last_became_active":"2023-12-16T21:13:36.503514+0000","last_became_peered":"2023-12-16T21:13:36.503514+0000","last_unstale":"2023-12-16T21:13:37.580539+0000","last_undegraded":"2023-12-16T21:13:37.580539+0000","last_fullsized":"2023-12-16T21:13:37.580539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:43:28.130579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.529137+0000","last_change":"2023-12-16T21:13:35.507575+0000","last_active":"2023-12-16T21:13:36.529137+0000","last_peered":"2023-12-16T21:13:36.529137+0000","last_clean":"2023-12-16T21:13:36.529137+0000","last_became_active":"2023-12-16T21:13:35.507291+0000","last_became_peered":"2023-12-16T21:13:35.507291+0000","last_unstale":"2023-12-16T21:13:36.529137+0000","last_undegraded":"2023-12-16T21:13:36.529137+0000","last_fullsized":"2023-12-16T21:13:36.529137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487138+0000","last_change":"2023-12-16T21:13:36.507654+0000","last_active":"2023-12-16T21:13:37.487138+0000","last_peered":"2023-12-16T21:13:37.487138+0000","last_clean":"2023-12-16T21:13:37.487138+0000","last_became_active":"2023-12-16T21:13:36.507245+0000","last_became_peered":"2023-12-16T21:13:36.507245+0000","last_unstale":"2023-12-16T21:13:37.487138+0000","last_undegraded":"2023-12-16T21:13:37.487138+0000","last_fullsized":"2023-12-16T21:13:37.487138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.487608+0000","last_change":"2023-12-16T21:13:35.509065+0000","last_active":"2023-12-16T21:13:36.487608+0000","last_peered":"2023-12-16T21:13:36.487608+0000","last_clean":"2023-12-16T21:13:36.487608+0000","last_became_active":"2023-12-16T21:13:35.508808+0000","last_became_peered":"2023-12-16T21:13:35.508808+0000","last_unstale":"2023-12-16T21:13:36.487608+0000","last_undegraded":"2023-12-16T21:13:36.487608+0000","last_fullsized":"2023-12-16T21:13:36.487608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580284+0000","last_change":"2023-12-16T21:13:36.507427+0000","last_active":"2023-12-16T21:13:37.580284+0000","last_peered":"2023-12-16T21:13:37.580284+0000","last_clean":"2023-12-16T21:13:37.580284+0000","last_became_active":"2023-12-16T21:13:36.507170+0000","last_became_peered":"2023-12-16T21:13:36.507170+0000","last_unstale":"2023-12-16T21:13:37.580284+0000","last_undegraded":"2023-12-16T21:13:37.580284+0000","last_fullsized":"2023-12-16T21:13:37.580284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490630+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:36.490630+0000","last_peered":"2023-12-16T21:13:36.490630+0000","last_clean":"2023-12-16T21:13:36.490630+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:36.490630+0000","last_undegraded":"2023-12-16T21:13:36.490630+0000","last_fullsized":"2023-12-16T21:13:36.490630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486371+0000","last_change":"2023-12-16T21:13:35.505945+0000","last_active":"2023-12-16T21:13:37.486371+0000","last_peered":"2023-12-16T21:13:37.486371+0000","last_clean":"2023-12-16T21:13:37.486371+0000","last_became_active":"2023-12-16T21:13:35.505670+0000","last_became_peered":"2023-12-16T21:13:35.505670+0000","last_unstale":"2023-12-16T21:13:37.486371+0000","last_undegraded":"2023-12-16T21:13:37.486371+0000","last_fullsized":"2023-12-16T21:13:37.486371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485341+0000","last_change":"2023-12-16T21:13:36.507596+0000","last_active":"2023-12-16T21:13:37.485341+0000","last_peered":"2023-12-16T21:13:37.485341+0000","last_clean":"2023-12-16T21:13:37.485341+0000","last_became_active":"2023-12-16T21:13:36.507302+0000","last_became_peered":"2023-12-16T21:13:36.507302+0000","last_unstale":"2023-12-16T21:13:37.485341+0000","last_undegraded":"2023-12-16T21:13:37.485341+0000","last_fullsized":"2023-12-16T21:13:37.485341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579838+0000","last_change":"2023-12-16T21:13:35.512856+0000","last_active":"2023-12-16T21:13:37.579838+0000","last_peered":"2023-12-16T21:13:37.579838+0000","last_clean":"2023-12-16T21:13:37.579838+0000","last_became_active":"2023-12-16T21:13:35.512539+0000","last_became_peered":"2023-12-16T21:13:35.512539+0000","last_unstale":"2023-12-16T21:13:37.579838+0000","last_undegraded":"2023-12-16T21:13:37.579838+0000","last_fullsized":"2023-12-16T21:13:37.579838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.504659+0000","last_change":"2023-12-16T21:13:36.503724+0000","last_active":"2023-12-16T21:13:36.504659+0000","last_peered":"2023-12-16T21:13:36.504659+0000","last_clean":"2023-12-16T21:13:36.504659+0000","last_became_active":"2023-12-16T21:13:36.503392+0000","last_became_peered":"2023-12-16T21:13:36.503392+0000","last_unstale":"2023-12-16T21:13:36.504659+0000","last_undegraded":"2023-12-16T21:13:36.504659+0000","last_fullsized":"2023-12-16T21:13:36.504659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:31:34.903149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.487011+0000","last_change":"2023-12-16T21:13:35.506777+0000","last_active":"2023-12-16T21:13:36.487011+0000","last_peered":"2023-12-16T21:13:36.487011+0000","last_clean":"2023-12-16T21:13:36.487011+0000","last_became_active":"2023-12-16T21:13:35.506483+0000","last_became_peered":"2023-12-16T21:13:35.506483+0000","last_unstale":"2023-12-16T21:13:36.487011+0000","last_undegraded":"2023-12-16T21:13:36.487011+0000","last_fullsized":"2023-12-16T21:13:36.487011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:25.556208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.497331+0000","last_change":"2023-12-16T21:13:36.496004+0000","last_active":"2023-12-16T21:13:36.497331+0000","last_peered":"2023-12-16T21:13:36.497331+0000","last_clean":"2023-12-16T21:13:36.497331+0000","last_became_active":"2023-12-16T21:13:36.495712+0000","last_became_peered":"2023-12-16T21:13:36.495712+0000","last_unstale":"2023-12-16T21:13:36.497331+0000","last_undegraded":"2023-12-16T21:13:36.497331+0000","last_fullsized":"2023-12-16T21:13:36.497331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486820+0000","last_change":"2023-12-16T21:13:35.507557+0000","last_active":"2023-12-16T21:13:37.486820+0000","last_peered":"2023-12-16T21:13:37.486820+0000","last_clean":"2023-12-16T21:13:37.486820+0000","last_became_active":"2023-12-16T21:13:35.507175+0000","last_became_peered":"2023-12-16T21:13:35.507175+0000","last_unstale":"2023-12-16T21:13:37.486820+0000","last_undegraded":"2023-12-16T21:13:37.486820+0000","last_fullsized":"2023-12-16T21:13:37.486820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:11.664911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.511648+0000","last_change":"2023-12-16T21:13:36.502502+0000","last_active":"2023-12-16T21:13:36.511648+0000","last_peered":"2023-12-16T21:13:36.511648+0000","last_clean":"2023-12-16T21:13:36.511648+0000","last_became_active":"2023-12-16T21:13:36.501893+0000","last_became_peered":"2023-12-16T21:13:36.501893+0000","last_unstale":"2023-12-16T21:13:36.511648+0000","last_undegraded":"2023-12-16T21:13:36.511648+0000","last_fullsized":"2023-12-16T21:13:36.511648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:53.131591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490728+0000","last_change":"2023-12-16T21:13:35.513556+0000","last_active":"2023-12-16T21:13:36.490728+0000","last_peered":"2023-12-16T21:13:36.490728+0000","last_clean":"2023-12-16T21:13:36.490728+0000","last_became_active":"2023-12-16T21:13:35.513208+0000","last_became_peered":"2023-12-16T21:13:35.513208+0000","last_unstale":"2023-12-16T21:13:36.490728+0000","last_undegraded":"2023-12-16T21:13:36.490728+0000","last_fullsized":"2023-12-16T21:13:36.490728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486956+0000","last_change":"2023-12-16T21:13:36.504553+0000","last_active":"2023-12-16T21:13:37.486956+0000","last_peered":"2023-12-16T21:13:37.486956+0000","last_clean":"2023-12-16T21:13:37.486956+0000","last_became_active":"2023-12-16T21:13:36.504276+0000","last_became_peered":"2023-12-16T21:13:36.504276+0000","last_unstale":"2023-12-16T21:13:37.486956+0000","last_undegraded":"2023-12-16T21:13:37.486956+0000","last_fullsized":"2023-12-16T21:13:37.486956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580388+0000","last_change":"2023-12-16T21:13:35.511628+0000","last_active":"2023-12-16T21:13:37.580388+0000","last_peered":"2023-12-16T21:13:37.580388+0000","last_clean":"2023-12-16T21:13:37.580388+0000","last_became_active":"2023-12-16T21:13:35.510934+0000","last_became_peered":"2023-12-16T21:13:35.510934+0000","last_unstale":"2023-12-16T21:13:37.580388+0000","last_undegraded":"2023-12-16T21:13:37.580388+0000","last_fullsized":"2023-12-16T21:13:37.580388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:21.664872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.497338+0000","last_change":"2023-12-16T21:13:36.495330+0000","last_active":"2023-12-16T21:13:36.497338+0000","last_peered":"2023-12-16T21:13:36.497338+0000","last_clean":"2023-12-16T21:13:36.497338+0000","last_became_active":"2023-12-16T21:13:36.495064+0000","last_became_peered":"2023-12-16T21:13:36.495064+0000","last_unstale":"2023-12-16T21:13:36.497338+0000","last_undegraded":"2023-12-16T21:13:36.497338+0000","last_fullsized":"2023-12-16T21:13:36.497338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.493340+0000","last_change":"2023-12-16T21:13:35.508842+0000","last_active":"2023-12-16T21:13:36.493340+0000","last_peered":"2023-12-16T21:13:36.493340+0000","last_clean":"2023-12-16T21:13:36.493340+0000","last_became_active":"2023-12-16T21:13:35.508552+0000","last_became_peered":"2023-12-16T21:13:35.508552+0000","last_unstale":"2023-12-16T21:13:36.493340+0000","last_undegraded":"2023-12-16T21:13:36.493340+0000","last_fullsized":"2023-12-16T21:13:36.493340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.497136+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.496859+0000","last_became_peered":"2023-12-16T21:13:36.496859+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486007+0000","last_change":"2023-12-16T21:13:35.512677+0000","last_active":"2023-12-16T21:13:37.486007+0000","last_peered":"2023-12-16T21:13:37.486007+0000","last_clean":"2023-12-16T21:13:37.486007+0000","last_became_active":"2023-12-16T21:13:35.511936+0000","last_became_peered":"2023-12-16T21:13:35.511936+0000","last_unstale":"2023-12-16T21:13:37.486007+0000","last_undegraded":"2023-12-16T21:13:37.486007+0000","last_fullsized":"2023-12-16T21:13:37.486007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501183+0000","last_change":"2023-12-16T21:13:36.499822+0000","last_active":"2023-12-16T21:13:36.501183+0000","last_peered":"2023-12-16T21:13:36.501183+0000","last_clean":"2023-12-16T21:13:36.501183+0000","last_became_active":"2023-12-16T21:13:36.499517+0000","last_became_peered":"2023-12-16T21:13:36.499517+0000","last_unstale":"2023-12-16T21:13:36.501183+0000","last_undegraded":"2023-12-16T21:13:36.501183+0000","last_fullsized":"2023-12-16T21:13:36.501183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:47.663889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579837+0000","last_change":"2023-12-16T21:13:35.511783+0000","last_active":"2023-12-16T21:13:37.579837+0000","last_peered":"2023-12-16T21:13:37.579837+0000","last_clean":"2023-12-16T21:13:37.579837+0000","last_became_active":"2023-12-16T21:13:35.510752+0000","last_became_peered":"2023-12-16T21:13:35.510752+0000","last_unstale":"2023-12-16T21:13:37.579837+0000","last_undegraded":"2023-12-16T21:13:37.579837+0000","last_fullsized":"2023-12-16T21:13:37.579837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.511735+0000","last_change":"2023-12-16T21:13:36.503029+0000","last_active":"2023-12-16T21:13:36.511735+0000","last_peered":"2023-12-16T21:13:36.511735+0000","last_clean":"2023-12-16T21:13:36.511735+0000","last_became_active":"2023-12-16T21:13:36.502641+0000","last_became_peered":"2023-12-16T21:13:36.502641+0000","last_unstale":"2023-12-16T21:13:36.511735+0000","last_undegraded":"2023-12-16T21:13:36.511735+0000","last_fullsized":"2023-12-16T21:13:36.511735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:26.560123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.494285+0000","last_change":"2023-12-16T21:13:35.511492+0000","last_active":"2023-12-16T21:13:36.494285+0000","last_peered":"2023-12-16T21:13:36.494285+0000","last_clean":"2023-12-16T21:13:36.494285+0000","last_became_active":"2023-12-16T21:13:35.511118+0000","last_became_peered":"2023-12-16T21:13:35.511118+0000","last_unstale":"2023-12-16T21:13:36.494285+0000","last_undegraded":"2023-12-16T21:13:36.494285+0000","last_fullsized":"2023-12-16T21:13:36.494285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485112+0000","last_change":"2023-12-16T21:13:36.495196+0000","last_active":"2023-12-16T21:13:37.485112+0000","last_peered":"2023-12-16T21:13:37.485112+0000","last_clean":"2023-12-16T21:13:37.485112+0000","last_became_active":"2023-12-16T21:13:36.494933+0000","last_became_peered":"2023-12-16T21:13:36.494933+0000","last_unstale":"2023-12-16T21:13:37.485112+0000","last_undegraded":"2023-12-16T21:13:37.485112+0000","last_fullsized":"2023-12-16T21:13:37.485112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.494120+0000","last_change":"2023-12-16T21:13:35.506652+0000","last_active":"2023-12-16T21:13:36.494120+0000","last_peered":"2023-12-16T21:13:36.494120+0000","last_clean":"2023-12-16T21:13:36.494120+0000","last_became_active":"2023-12-16T21:13:35.506349+0000","last_became_peered":"2023-12-16T21:13:35.506349+0000","last_unstale":"2023-12-16T21:13:36.494120+0000","last_undegraded":"2023-12-16T21:13:36.494120+0000","last_fullsized":"2023-12-16T21:13:36.494120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485338+0000","last_change":"2023-12-16T21:13:36.490929+0000","last_active":"2023-12-16T21:13:37.485338+0000","last_peered":"2023-12-16T21:13:37.485338+0000","last_clean":"2023-12-16T21:13:37.485338+0000","last_became_active":"2023-12-16T21:13:36.490557+0000","last_became_peered":"2023-12-16T21:13:36.490557+0000","last_unstale":"2023-12-16T21:13:37.485338+0000","last_undegraded":"2023-12-16T21:13:37.485338+0000","last_fullsized":"2023-12-16T21:13:37.485338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580200+0000","last_change":"2023-12-16T21:13:35.511062+0000","last_active":"2023-12-16T21:13:37.580200+0000","last_peered":"2023-12-16T21:13:37.580200+0000","last_clean":"2023-12-16T21:13:37.580200+0000","last_became_active":"2023-12-16T21:13:35.510780+0000","last_became_peered":"2023-12-16T21:13:35.510780+0000","last_unstale":"2023-12-16T21:13:37.580200+0000","last_undegraded":"2023-12-16T21:13:37.580200+0000","last_fullsized":"2023-12-16T21:13:37.580200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:25.707555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485439+0000","last_change":"2023-12-16T21:13:36.499546+0000","last_active":"2023-12-16T21:13:37.485439+0000","last_peered":"2023-12-16T21:13:37.485439+0000","last_clean":"2023-12-16T21:13:37.485439+0000","last_became_active":"2023-12-16T21:13:36.499236+0000","last_became_peered":"2023-12-16T21:13:36.499236+0000","last_unstale":"2023-12-16T21:13:37.485439+0000","last_undegraded":"2023-12-16T21:13:37.485439+0000","last_fullsized":"2023-12-16T21:13:37.485439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486745+0000","last_change":"2023-12-16T21:13:36.512546+0000","last_active":"2023-12-16T21:13:37.486745+0000","last_peered":"2023-12-16T21:13:37.486745+0000","last_clean":"2023-12-16T21:13:37.486745+0000","last_became_active":"2023-12-16T21:13:36.512196+0000","last_became_peered":"2023-12-16T21:13:36.512196+0000","last_unstale":"2023-12-16T21:13:37.486745+0000","last_undegraded":"2023-12-16T21:13:37.486745+0000","last_fullsized":"2023-12-16T21:13:37.486745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580134+0000","last_change":"2023-12-16T21:13:35.511999+0000","last_active":"2023-12-16T21:13:37.580134+0000","last_peered":"2023-12-16T21:13:37.580134+0000","last_clean":"2023-12-16T21:13:37.580134+0000","last_became_active":"2023-12-16T21:13:35.511760+0000","last_became_peered":"2023-12-16T21:13:35.511760+0000","last_unstale":"2023-12-16T21:13:37.580134+0000","last_undegraded":"2023-12-16T21:13:37.580134+0000","last_fullsized":"2023-12-16T21:13:37.580134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:49.269497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487061+0000","last_change":"2023-12-16T21:13:36.512673+0000","last_active":"2023-12-16T21:13:37.487061+0000","last_peered":"2023-12-16T21:13:37.487061+0000","last_clean":"2023-12-16T21:13:37.487061+0000","last_became_active":"2023-12-16T21:13:36.512185+0000","last_became_peered":"2023-12-16T21:13:36.512185+0000","last_unstale":"2023-12-16T21:13:37.487061+0000","last_undegraded":"2023-12-16T21:13:37.487061+0000","last_fullsized":"2023-12-16T21:13:37.487061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580255+0000","last_change":"2023-12-16T21:13:35.511982+0000","last_active":"2023-12-16T21:13:37.580255+0000","last_peered":"2023-12-16T21:13:37.580255+0000","last_clean":"2023-12-16T21:13:37.580255+0000","last_became_active":"2023-12-16T21:13:35.511359+0000","last_became_peered":"2023-12-16T21:13:35.511359+0000","last_unstale":"2023-12-16T21:13:37.580255+0000","last_undegraded":"2023-12-16T21:13:37.580255+0000","last_fullsized":"2023-12-16T21:13:37.580255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.506611+0000","last_change":"2023-12-16T21:13:36.505890+0000","last_active":"2023-12-16T21:13:36.506611+0000","last_peered":"2023-12-16T21:13:36.506611+0000","last_clean":"2023-12-16T21:13:36.506611+0000","last_became_active":"2023-12-16T21:13:36.505414+0000","last_became_peered":"2023-12-16T21:13:36.505414+0000","last_unstale":"2023-12-16T21:13:36.506611+0000","last_undegraded":"2023-12-16T21:13:36.506611+0000","last_fullsized":"2023-12-16T21:13:36.506611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485644+0000","last_change":"2023-12-16T21:13:35.511468+0000","last_active":"2023-12-16T21:13:37.485644+0000","last_peered":"2023-12-16T21:13:37.485644+0000","last_clean":"2023-12-16T21:13:37.485644+0000","last_became_active":"2023-12-16T21:13:35.511092+0000","last_became_peered":"2023-12-16T21:13:35.511092+0000","last_unstale":"2023-12-16T21:13:37.485644+0000","last_undegraded":"2023-12-16T21:13:37.485644+0000","last_fullsized":"2023-12-16T21:13:37.485644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485618+0000","last_change":"2023-12-16T21:13:36.494416+0000","last_active":"2023-12-16T21:13:37.485618+0000","last_peered":"2023-12-16T21:13:37.485618+0000","last_clean":"2023-12-16T21:13:37.485618+0000","last_became_active":"2023-12-16T21:13:36.494121+0000","last_became_peered":"2023-12-16T21:13:36.494121+0000","last_unstale":"2023-12-16T21:13:37.485618+0000","last_undegraded":"2023-12-16T21:13:37.485618+0000","last_fullsized":"2023-12-16T21:13:37.485618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.529329+0000","last_change":"2023-12-16T21:13:35.510139+0000","last_active":"2023-12-16T21:13:36.529329+0000","last_peered":"2023-12-16T21:13:36.529329+0000","last_clean":"2023-12-16T21:13:36.529329+0000","last_became_active":"2023-12-16T21:13:35.509804+0000","last_became_peered":"2023-12-16T21:13:35.509804+0000","last_unstale":"2023-12-16T21:13:36.529329+0000","last_undegraded":"2023-12-16T21:13:36.529329+0000","last_fullsized":"2023-12-16T21:13:36.529329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:56:59.813907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":69407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:40.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-12-16T21:13:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.367+0000 7f734525c640 1 Processor -- start 2023-12-16T21:13:40.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.367+0000 7f734525c640 1 -- start start 2023-12-16T21:13:40.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f734525c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f734015d330 0x7f73400b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f734525c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73400d2d90 0x7f73400b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f734525c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400b2b10 con 0x7f73400d2910 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f734525c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400d1590 con 0x7f73400d2d90 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f734525c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400d1870 con 0x7f734015d330 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f733f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73400d2d90 0x7f73400b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f733f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73400d2d90 0x7f73400b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54472/0 (socket says 172.21.15.46:54472) 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f733f59e640 1 -- 172.21.15.46:0/3820328908 learned_addr learned my addr 172.21.15.46:0/3820328908 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.368+0000 7f733ed9d640 1 --2- 172.21.15.46:0/3820328908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f734015d330 0x7f73400b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.369+0000 7f733f59e640 1 -- 172.21.15.46:0/3820328908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f734015d330 msgr2=0x7f73400b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.369+0000 7f733f59e640 1 --2- 172.21.15.46:0/3820328908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f734015d330 0x7f73400b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.369+0000 7f733f59e640 1 -- 172.21.15.46:0/3820328908 >> v1:172.21.15.121:6789/0 conn(0x7f73400d2910 legacy=0x7f73400d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:40.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.369+0000 7f733f59e640 1 -- 172.21.15.46:0/3820328908 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400d1b50 con 0x7f73400d2d90 2023-12-16T21:13:40.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.369+0000 7f733f59e640 1 --2- 172.21.15.46:0/3820328908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400d2d90 0x7f73400b6390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7330009e50 tx=0x7f733002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93b189148ab6058 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f733dd9b640 1 -- 172.21.15.46:0/3820328908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733002ca70 con 0x7f73400d2d90 2023-12-16T21:13:40.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f733dd9b640 1 -- 172.21.15.46:0/3820328908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733002cbd0 con 0x7f73400d2d90 2023-12-16T21:13:40.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f733dd9b640 1 -- 172.21.15.46:0/3820328908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73300343f0 con 0x7f73400d2d90 2023-12-16T21:13:40.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 -- 172.21.15.46:0/3820328908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400d2d90 msgr2=0x7f73400b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 --2- 172.21.15.46:0/3820328908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400d2d90 0x7f73400b6390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7330009e50 tx=0x7f733002bed0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 -- 172.21.15.46:0/3820328908 shutdown_connections 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 --2- 172.21.15.46:0/3820328908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73400d2d90 0x7f73400b6390 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 --2- 172.21.15.46:0/3820328908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f734015d330 0x7f73400b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 -- 172.21.15.46:0/3820328908 >> 172.21.15.46:0/3820328908 conn(0x7f7340072020 msgr2=0x7f7340072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 -- 172.21.15.46:0/3820328908 shutdown_connections 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.370+0000 7f734525c640 1 -- 172.21.15.46:0/3820328908 wait complete. 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 Processor -- start 2023-12-16T21:13:40.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 -- start start 2023-12-16T21:13:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73400d2910 0x7f73401f6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73400d2d90 0x7f73400cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 0x7f73400ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73400c7bf0 con 0x7f734015d330 2023-12-16T21:13:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73400ccbc0 con 0x7f73400d2d90 2023-12-16T21:13:40.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f734525c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73400ccea0 con 0x7f73400d2910 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.372+0000 7f733f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 0x7f73400ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 0x7f73400ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54484/0 (socket says 172.21.15.46:54484) 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 -- 172.21.15.46:0/2366072609 learned_addr learned my addr 172.21.15.46:0/2366072609 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733ed9d640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73400d2910 0x7f73401f6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733e59c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73400d2d90 0x7f73400cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 -- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73400d2910 msgr2=0x7f73401f6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73400d2910 0x7f73401f6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 -- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73400d2d90 msgr2=0x7f73400cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73400d2d90 0x7f73400cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 -- 172.21.15.46:0/2366072609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7330009b30 con 0x7f734015d330 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f733f59e640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 0x7f73400ca330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7330009e50 tx=0x7f7330004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330009510 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733003e450 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73400cd180 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733003e800 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.373+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73400cc7b0 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.374+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73400c7da0 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.375+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f733003e960 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.375+0000 7f73377fe640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f732006ae40 0x7f732006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.375+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73300ac220 con 0x7f734015d330 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.375+0000 7f733ed9d640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f732006ae40 0x7f732006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.376+0000 7f733ed9d640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f732006ae40 0x7f732006d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f73380079a0 tx=0x7f7338008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.379+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f73400c7da0 con 0x7f734015d330 2023-12-16T21:13:40.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.625+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f73400cc970 con 0x7f734015d330 2023-12-16T21:13:40.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.625+0000 7f73377fe640 1 -- 172.21.15.46:0/2366072609 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1010 (secure 0 0 0) 0x7f7330077380 con 0x7f734015d330 2023-12-16T21:13:40.625 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:40.625 INFO:teuthology.orchestra.run.smithi046.stdout:{"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":28,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":21,"num_osds":8,"num_up_osds":8,"osd_up_since":1702761202,"num_in_osds":8,"osd_in_since":1702761193,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":241319936,"bytes_avail":772852793344,"bytes_total":773094113280,"write_bytes_sec":2621,"read_op_per_sec":0,"write_op_per_sec":8},"fsmap":{"epoch":6,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"d","status":"up:active","gid":4294}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-12-16T21:13:11.739725+0000","services":{}},"progress_events":{}} 2023-12-16T21:13:40.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f732006ae40 msgr2=0x7f732006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f732006ae40 0x7f732006d300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f73380079a0 tx=0x7f7338008040 comp rx=0 tx=0).stop 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 msgr2=0x7f73400ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 0x7f73400ca330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7330009e50 tx=0x7f7330004a40 comp rx=0 tx=0).stop 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 shutdown_connections 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f732006ae40 0x7f732006d300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f734015d330 0x7f73400ca330 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73400d2d90 0x7f73400cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 --2- 172.21.15.46:0/2366072609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73400d2910 0x7f73401f6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.627+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 >> 172.21.15.46:0/2366072609 conn(0x7f7340072020 msgr2=0x7f73400cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.628+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 shutdown_connections 2023-12-16T21:13:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.628+0000 7f734525c640 1 -- 172.21.15.46:0/2366072609 wait complete. 2023-12-16T21:13:40.638 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-12-16T21:13:40.638 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:40.745 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.746+0000 7f0b4ee89640 1 Processor -- start 2023-12-16T21:13:40.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.746+0000 7f0b4ee89640 1 -- start start 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4ee89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b4815d2d0 0x7f0b480b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4ee89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b480d1bb0 con 0x7f0b480d2040 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4ee89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4815d6b0 con 0x7f0b480ce150 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4ee89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b480ce700 con 0x7f0b4815d2d0 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4cc25640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0b4815d2d0 0x7f0b480b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b47fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0b480d2040 0x7f0b480cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53422/0 (socket says 172.21.15.46:53422) 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b47fff640 1 -- 172.21.15.46:0/2185919856 learned_addr learned my addr 172.21.15.46:0/2185919856 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:40.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4cc25640 1 -- 172.21.15.46:0/2185919856 >> v1:172.21.15.121:6789/0 conn(0x7f0b480ce150 legacy=0x7f0b480b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:40.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4cc25640 1 -- 172.21.15.46:0/2185919856 >> v1:172.21.15.46:6789/0 conn(0x7f0b480d2040 legacy=0x7f0b480cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b477fe640 1 -- 172.21.15.46:0/2185919856 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363487311 0 0) 0x7f0b480d1bb0 con 0x7f0b480d2040 2023-12-16T21:13:40.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.747+0000 7f0b4cc25640 1 -- 172.21.15.46:0/2185919856 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480d1280 con 0x7f0b4815d2d0 2023-12-16T21:13:40.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.748+0000 7f0b4cc25640 1 --2- 172.21.15.46:0/2185919856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4815d2d0 0x7f0b480b4c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0097a0 tx=0x7f0b3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb35d6bbaa751391 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.748+0000 7f0b477fe640 1 -- 172.21.15.46:0/2185919856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c031830 con 0x7f0b4815d2d0 2023-12-16T21:13:40.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.748+0000 7f0b477fe640 1 -- 172.21.15.46:0/2185919856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c031990 con 0x7f0b4815d2d0 2023-12-16T21:13:40.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.748+0000 7f0b477fe640 1 -- 172.21.15.46:0/2185919856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c031d20 con 0x7f0b4815d2d0 2023-12-16T21:13:40.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 -- 172.21.15.46:0/2185919856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4815d2d0 msgr2=0x7f0b480b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/2185919856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4815d2d0 0x7f0b480b4c60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c0097a0 tx=0x7f0b3c02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 -- 172.21.15.46:0/2185919856 shutdown_connections 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/2185919856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b4815d2d0 0x7f0b480b4c60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 -- 172.21.15.46:0/2185919856 >> 172.21.15.46:0/2185919856 conn(0x7f0b48072050 msgr2=0x7f0b48072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 -- 172.21.15.46:0/2185919856 shutdown_connections 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.749+0000 7f0b4ee89640 1 -- 172.21.15.46:0/2185919856 wait complete. 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.750+0000 7f0b4ee89640 1 Processor -- start 2023-12-16T21:13:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.750+0000 7f0b4ee89640 1 -- start start 2023-12-16T21:13:40.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4ee89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 0x7f0b480c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4ee89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b480d2040 0x7f0b480c6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4ee89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b4815d2d0 0x7f0b480b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4ee89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480c7d10 con 0x7f0b480ce150 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4ee89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480c66b0 con 0x7f0b480d2040 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4ee89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480ccfe0 con 0x7f0b4815d2d0 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 0x7f0b480c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 0x7f0b480c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36044/0 (socket says 172.21.15.46:36044) 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 -- 172.21.15.46:0/3380547271 learned_addr learned my addr 172.21.15.46:0/3380547271 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4d426640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b4815d2d0 0x7f0b480b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b47fff640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b480d2040 0x7f0b480c6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 -- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b4815d2d0 msgr2=0x7f0b480b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b4815d2d0 0x7f0b480b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 -- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b480d2040 msgr2=0x7f0b480c6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b480d2040 0x7f0b480c6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 -- 172.21.15.46:0/3380547271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c009400 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4d426640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b4815d2d0 0x7f0b480b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b47fff640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b480d2040 0x7f0b480c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.751+0000 7f0b4cc25640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 0x7f0b480c77d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c000c00 tx=0x7f0b3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.752+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c03aa60 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.752+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c03abc0 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.752+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b481f7730 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.752+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c0443f0 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.752+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b480c6cf0 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.753+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48069400 con 0x7f0b480ce150 2023-12-16T21:13:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.753+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0b3c04d450 con 0x7f0b480ce150 2023-12-16T21:13:40.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.753+0000 7f0b457fa640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0b2c06ae40 0x7f0b2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:40.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.754+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b3c0ab7f0 con 0x7f0b480ce150 2023-12-16T21:13:40.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.754+0000 7f0b47fff640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0b2c06ae40 0x7f0b2c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:40.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.754+0000 7f0b47fff640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0b2c06ae40 0x7f0b2c06d300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b4815dc40 tx=0x7f0b34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:40.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.758+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0b3c0768a0 con 0x7f0b480ce150 2023-12-16T21:13:40.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.933+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b480d1840 con 0x7f0b2c06ae40 2023-12-16T21:13:40.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.940+0000 7f0b457fa640 1 -- 172.21.15.46:0/3380547271 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264676 (secure 0 0 0) 0x7f0b340013d0 con 0x7f0b2c06ae40 2023-12-16T21:13:40.943 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:40.944 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:40.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0b2c06ae40 msgr2=0x7f0b2c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0b2c06ae40 0x7f0b2c06d300 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b4815dc40 tx=0x7f0b34009040 comp rx=0 tx=0).stop 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 msgr2=0x7f0b480c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 0x7f0b480c77d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c000c00 tx=0x7f0b3c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 shutdown_connections 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0b2c06ae40 0x7f0b2c06d300 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b4815d2d0 0x7f0b480b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b480d2040 0x7f0b480c6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 --2- 172.21.15.46:0/3380547271 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b480ce150 0x7f0b480c77d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 >> 172.21.15.46:0/3380547271 conn(0x7f0b48072050 msgr2=0x7f0b4815e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 shutdown_connections 2023-12-16T21:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:40.943+0000 7f0b4ee89640 1 -- 172.21.15.46:0/3380547271 wait complete. 2023-12-16T21:13:40.953 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-16T21:13:39.955609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235664,"kb_used_data":2836,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739056,"statfs":{"total":773094113280,"available":772852793344,"internally_reserved":0,"allocated":2904064,"data_stored":1746104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.467658"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580448+0000","last_change":"2023-12-16T21:13:36.497094+0000","last_active":"2023-12-16T21:13:37.580448+0000","last_peered":"2023-12-16T21:13:37.580448+0000","last_clean":"2023-12-16T21:13:37.580448+0000","last_became_active":"2023-12-16T21:13:36.496839+0000","last_became_peered":"2023-12-16T21:13:36.496839+0000","last_unstale":"2023-12-16T21:13:37.580448+0000","last_undegraded":"2023-12-16T21:13:37.580448+0000","last_fullsized":"2023-12-16T21:13:37.580448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.499656+0000","last_change":"2023-12-16T21:13:36.496641+0000","last_active":"2023-12-16T21:13:36.499656+0000","last_peered":"2023-12-16T21:13:36.499656+0000","last_clean":"2023-12-16T21:13:36.499656+0000","last_became_active":"2023-12-16T21:13:36.496301+0000","last_became_peered":"2023-12-16T21:13:36.496301+0000","last_unstale":"2023-12-16T21:13:36.499656+0000","last_undegraded":"2023-12-16T21:13:36.499656+0000","last_fullsized":"2023-12-16T21:13:36.499656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:57:00.817822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.491825+0000","last_change":"2023-12-16T21:13:36.490436+0000","last_active":"2023-12-16T21:13:36.491825+0000","last_peered":"2023-12-16T21:13:36.491825+0000","last_clean":"2023-12-16T21:13:36.491825+0000","last_became_active":"2023-12-16T21:13:36.489881+0000","last_became_peered":"2023-12-16T21:13:36.489881+0000","last_unstale":"2023-12-16T21:13:36.491825+0000","last_undegraded":"2023-12-16T21:13:36.491825+0000","last_fullsized":"2023-12-16T21:13:36.491825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:06:58.471822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487045+0000","last_change":"2023-12-16T21:13:36.507326+0000","last_active":"2023-12-16T21:13:37.487045+0000","last_peered":"2023-12-16T21:13:37.487045+0000","last_clean":"2023-12-16T21:13:37.487045+0000","last_became_active":"2023-12-16T21:13:36.507063+0000","last_became_peered":"2023-12-16T21:13:36.507063+0000","last_unstale":"2023-12-16T21:13:37.487045+0000","last_undegraded":"2023-12-16T21:13:37.487045+0000","last_fullsized":"2023-12-16T21:13:37.487045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580048+0000","last_change":"2023-12-16T21:13:36.494596+0000","last_active":"2023-12-16T21:13:37.580048+0000","last_peered":"2023-12-16T21:13:37.580048+0000","last_clean":"2023-12-16T21:13:37.580048+0000","last_became_active":"2023-12-16T21:13:36.494238+0000","last_became_peered":"2023-12-16T21:13:36.494238+0000","last_unstale":"2023-12-16T21:13:37.580048+0000","last_undegraded":"2023-12-16T21:13:37.580048+0000","last_fullsized":"2023-12-16T21:13:37.580048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486248+0000","last_change":"2023-12-16T21:13:36.505505+0000","last_active":"2023-12-16T21:13:37.486248+0000","last_peered":"2023-12-16T21:13:37.486248+0000","last_clean":"2023-12-16T21:13:37.486248+0000","last_became_active":"2023-12-16T21:13:36.505205+0000","last_became_peered":"2023-12-16T21:13:36.505205+0000","last_unstale":"2023-12-16T21:13:37.486248+0000","last_undegraded":"2023-12-16T21:13:37.486248+0000","last_fullsized":"2023-12-16T21:13:37.486248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579992+0000","last_change":"2023-12-16T21:13:36.502009+0000","last_active":"2023-12-16T21:13:37.579992+0000","last_peered":"2023-12-16T21:13:37.579992+0000","last_clean":"2023-12-16T21:13:37.579992+0000","last_became_active":"2023-12-16T21:13:36.501672+0000","last_became_peered":"2023-12-16T21:13:36.501672+0000","last_unstale":"2023-12-16T21:13:37.579992+0000","last_undegraded":"2023-12-16T21:13:37.579992+0000","last_fullsized":"2023-12-16T21:13:37.579992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.484971+0000","last_change":"2023-12-16T21:13:36.502551+0000","last_active":"2023-12-16T21:13:37.484971+0000","last_peered":"2023-12-16T21:13:37.484971+0000","last_clean":"2023-12-16T21:13:37.484971+0000","last_became_active":"2023-12-16T21:13:36.502214+0000","last_became_peered":"2023-12-16T21:13:36.502214+0000","last_unstale":"2023-12-16T21:13:37.484971+0000","last_undegraded":"2023-12-16T21:13:37.484971+0000","last_fullsized":"2023-12-16T21:13:37.484971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580474+0000","last_change":"2023-12-16T21:13:36.507546+0000","last_active":"2023-12-16T21:13:37.580474+0000","last_peered":"2023-12-16T21:13:37.580474+0000","last_clean":"2023-12-16T21:13:37.580474+0000","last_became_active":"2023-12-16T21:13:36.507253+0000","last_became_peered":"2023-12-16T21:13:36.507253+0000","last_unstale":"2023-12-16T21:13:37.580474+0000","last_undegraded":"2023-12-16T21:13:37.580474+0000","last_fullsized":"2023-12-16T21:13:37.580474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:22:24.104289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485901+0000","last_change":"2023-12-16T21:13:36.508602+0000","last_active":"2023-12-16T21:13:37.485901+0000","last_peered":"2023-12-16T21:13:37.485901+0000","last_clean":"2023-12-16T21:13:37.485901+0000","last_became_active":"2023-12-16T21:13:36.508223+0000","last_became_peered":"2023-12-16T21:13:36.508223+0000","last_unstale":"2023-12-16T21:13:37.485901+0000","last_undegraded":"2023-12-16T21:13:37.485901+0000","last_fullsized":"2023-12-16T21:13:37.485901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485806+0000","last_change":"2023-12-16T21:13:36.508624+0000","last_active":"2023-12-16T21:13:37.485806+0000","last_peered":"2023-12-16T21:13:37.485806+0000","last_clean":"2023-12-16T21:13:37.485806+0000","last_became_active":"2023-12-16T21:13:36.508293+0000","last_became_peered":"2023-12-16T21:13:36.508293+0000","last_unstale":"2023-12-16T21:13:37.485806+0000","last_undegraded":"2023-12-16T21:13:37.485806+0000","last_fullsized":"2023-12-16T21:13:37.485806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.509037+0000","last_change":"2023-12-16T21:13:36.499195+0000","last_active":"2023-12-16T21:13:36.509037+0000","last_peered":"2023-12-16T21:13:36.509037+0000","last_clean":"2023-12-16T21:13:36.509037+0000","last_became_active":"2023-12-16T21:13:36.498927+0000","last_became_peered":"2023-12-16T21:13:36.498927+0000","last_unstale":"2023-12-16T21:13:36.509037+0000","last_undegraded":"2023-12-16T21:13:36.509037+0000","last_fullsized":"2023-12-16T21:13:36.509037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:31:34.903149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485287+0000","last_change":"2023-12-16T21:13:36.494278+0000","last_active":"2023-12-16T21:13:37.485287+0000","last_peered":"2023-12-16T21:13:37.485287+0000","last_clean":"2023-12-16T21:13:37.485287+0000","last_became_active":"2023-12-16T21:13:36.493970+0000","last_became_peered":"2023-12-16T21:13:36.493970+0000","last_unstale":"2023-12-16T21:13:37.485287+0000","last_undegraded":"2023-12-16T21:13:37.485287+0000","last_fullsized":"2023-12-16T21:13:37.485287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501464+0000","last_change":"2023-12-16T21:13:36.499374+0000","last_active":"2023-12-16T21:13:36.501464+0000","last_peered":"2023-12-16T21:13:36.501464+0000","last_clean":"2023-12-16T21:13:36.501464+0000","last_became_active":"2023-12-16T21:13:36.499097+0000","last_became_peered":"2023-12-16T21:13:36.499097+0000","last_unstale":"2023-12-16T21:13:36.501464+0000","last_undegraded":"2023-12-16T21:13:36.501464+0000","last_fullsized":"2023-12-16T21:13:36.501464+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:53.131591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.505430+0000","last_change":"2023-12-16T21:13:36.504350+0000","last_active":"2023-12-16T21:13:36.505430+0000","last_peered":"2023-12-16T21:13:36.505430+0000","last_clean":"2023-12-16T21:13:36.505430+0000","last_became_active":"2023-12-16T21:13:36.504046+0000","last_became_peered":"2023-12-16T21:13:36.504046+0000","last_unstale":"2023-12-16T21:13:36.505430+0000","last_undegraded":"2023-12-16T21:13:36.505430+0000","last_fullsized":"2023-12-16T21:13:36.505430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485909+0000","last_change":"2023-12-16T21:13:36.502440+0000","last_active":"2023-12-16T21:13:37.485909+0000","last_peered":"2023-12-16T21:13:37.485909+0000","last_clean":"2023-12-16T21:13:37.485909+0000","last_became_active":"2023-12-16T21:13:36.502160+0000","last_became_peered":"2023-12-16T21:13:36.502160+0000","last_unstale":"2023-12-16T21:13:37.485909+0000","last_undegraded":"2023-12-16T21:13:37.485909+0000","last_fullsized":"2023-12-16T21:13:37.485909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487000+0000","last_change":"2023-12-16T21:13:36.508778+0000","last_active":"2023-12-16T21:13:37.487000+0000","last_peered":"2023-12-16T21:13:37.487000+0000","last_clean":"2023-12-16T21:13:37.487000+0000","last_became_active":"2023-12-16T21:13:36.508506+0000","last_became_peered":"2023-12-16T21:13:36.508506+0000","last_unstale":"2023-12-16T21:13:37.487000+0000","last_undegraded":"2023-12-16T21:13:37.487000+0000","last_fullsized":"2023-12-16T21:13:37.487000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.495755+0000","last_change":"2023-12-16T21:13:36.494459+0000","last_active":"2023-12-16T21:13:36.495755+0000","last_peered":"2023-12-16T21:13:36.495755+0000","last_clean":"2023-12-16T21:13:36.495755+0000","last_became_active":"2023-12-16T21:13:36.494156+0000","last_became_peered":"2023-12-16T21:13:36.494156+0000","last_unstale":"2023-12-16T21:13:36.495755+0000","last_undegraded":"2023-12-16T21:13:36.495755+0000","last_fullsized":"2023-12-16T21:13:36.495755+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:47.663889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580395+0000","last_change":"2023-12-16T21:13:36.496446+0000","last_active":"2023-12-16T21:13:37.580395+0000","last_peered":"2023-12-16T21:13:37.580395+0000","last_clean":"2023-12-16T21:13:37.580395+0000","last_became_active":"2023-12-16T21:13:36.496123+0000","last_became_peered":"2023-12-16T21:13:36.496123+0000","last_unstale":"2023-12-16T21:13:37.580395+0000","last_undegraded":"2023-12-16T21:13:37.580395+0000","last_fullsized":"2023-12-16T21:13:37.580395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486172+0000","last_change":"2023-12-16T21:13:36.505223+0000","last_active":"2023-12-16T21:13:37.486172+0000","last_peered":"2023-12-16T21:13:37.486172+0000","last_clean":"2023-12-16T21:13:37.486172+0000","last_became_active":"2023-12-16T21:13:36.504834+0000","last_became_peered":"2023-12-16T21:13:36.504834+0000","last_unstale":"2023-12-16T21:13:37.486172+0000","last_undegraded":"2023-12-16T21:13:37.486172+0000","last_fullsized":"2023-12-16T21:13:37.486172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501391+0000","last_change":"2023-12-16T21:13:36.500880+0000","last_active":"2023-12-16T21:13:36.501391+0000","last_peered":"2023-12-16T21:13:36.501391+0000","last_clean":"2023-12-16T21:13:36.501391+0000","last_became_active":"2023-12-16T21:13:36.500532+0000","last_became_peered":"2023-12-16T21:13:36.500532+0000","last_unstale":"2023-12-16T21:13:36.501391+0000","last_undegraded":"2023-12-16T21:13:36.501391+0000","last_fullsized":"2023-12-16T21:13:36.501391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486491+0000","last_change":"2023-12-16T21:13:36.507578+0000","last_active":"2023-12-16T21:13:37.486491+0000","last_peered":"2023-12-16T21:13:37.486491+0000","last_clean":"2023-12-16T21:13:37.486491+0000","last_became_active":"2023-12-16T21:13:36.507206+0000","last_became_peered":"2023-12-16T21:13:36.507206+0000","last_unstale":"2023-12-16T21:13:37.486491+0000","last_undegraded":"2023-12-16T21:13:37.486491+0000","last_fullsized":"2023-12-16T21:13:37.486491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485167+0000","last_change":"2023-12-16T21:13:36.506029+0000","last_active":"2023-12-16T21:13:37.485167+0000","last_peered":"2023-12-16T21:13:37.485167+0000","last_clean":"2023-12-16T21:13:37.485167+0000","last_became_active":"2023-12-16T21:13:36.505785+0000","last_became_peered":"2023-12-16T21:13:36.505785+0000","last_unstale":"2023-12-16T21:13:37.485167+0000","last_undegraded":"2023-12-16T21:13:37.485167+0000","last_fullsized":"2023-12-16T21:13:37.485167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486054+0000","last_change":"2023-12-16T21:13:36.504916+0000","last_active":"2023-12-16T21:13:37.486054+0000","last_peered":"2023-12-16T21:13:37.486054+0000","last_clean":"2023-12-16T21:13:37.486054+0000","last_became_active":"2023-12-16T21:13:36.504422+0000","last_became_peered":"2023-12-16T21:13:36.504422+0000","last_unstale":"2023-12-16T21:13:37.486054+0000","last_undegraded":"2023-12-16T21:13:37.486054+0000","last_fullsized":"2023-12-16T21:13:37.486054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.503543+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.503210+0000","last_became_peered":"2023-12-16T21:13:36.503210+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486065+0000","last_change":"2023-12-16T21:13:36.494611+0000","last_active":"2023-12-16T21:13:37.486065+0000","last_peered":"2023-12-16T21:13:37.486065+0000","last_clean":"2023-12-16T21:13:37.486065+0000","last_became_active":"2023-12-16T21:13:36.494373+0000","last_became_peered":"2023-12-16T21:13:36.494373+0000","last_unstale":"2023-12-16T21:13:37.486065+0000","last_undegraded":"2023-12-16T21:13:37.486065+0000","last_fullsized":"2023-12-16T21:13:37.486065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.500812+0000","last_change":"2023-12-16T21:13:36.498640+0000","last_active":"2023-12-16T21:13:36.500812+0000","last_peered":"2023-12-16T21:13:36.500812+0000","last_clean":"2023-12-16T21:13:36.500812+0000","last_became_active":"2023-12-16T21:13:36.498334+0000","last_became_peered":"2023-12-16T21:13:36.498334+0000","last_unstale":"2023-12-16T21:13:36.500812+0000","last_undegraded":"2023-12-16T21:13:36.500812+0000","last_fullsized":"2023-12-16T21:13:36.500812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485419+0000","last_change":"2023-12-16T21:13:36.506001+0000","last_active":"2023-12-16T21:13:37.485419+0000","last_peered":"2023-12-16T21:13:37.485419+0000","last_clean":"2023-12-16T21:13:37.485419+0000","last_became_active":"2023-12-16T21:13:36.505724+0000","last_became_peered":"2023-12-16T21:13:36.505724+0000","last_unstale":"2023-12-16T21:13:37.485419+0000","last_undegraded":"2023-12-16T21:13:37.485419+0000","last_fullsized":"2023-12-16T21:13:37.485419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.491115+0000","last_change":"2023-12-16T21:13:36.489974+0000","last_active":"2023-12-16T21:13:36.491115+0000","last_peered":"2023-12-16T21:13:36.491115+0000","last_clean":"2023-12-16T21:13:36.491115+0000","last_became_active":"2023-12-16T21:13:36.489597+0000","last_became_peered":"2023-12-16T21:13:36.489597+0000","last_unstale":"2023-12-16T21:13:36.491115+0000","last_undegraded":"2023-12-16T21:13:36.491115+0000","last_fullsized":"2023-12-16T21:13:36.491115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:46.012193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486570+0000","last_change":"2023-12-16T21:13:36.505535+0000","last_active":"2023-12-16T21:13:37.486570+0000","last_peered":"2023-12-16T21:13:37.486570+0000","last_clean":"2023-12-16T21:13:37.486570+0000","last_became_active":"2023-12-16T21:13:36.505185+0000","last_became_peered":"2023-12-16T21:13:36.505185+0000","last_unstale":"2023-12-16T21:13:37.486570+0000","last_undegraded":"2023-12-16T21:13:37.486570+0000","last_fullsized":"2023-12-16T21:13:37.486570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.505879+0000","last_change":"2023-12-16T21:13:36.504923+0000","last_active":"2023-12-16T21:13:36.505879+0000","last_peered":"2023-12-16T21:13:36.505879+0000","last_clean":"2023-12-16T21:13:36.505879+0000","last_became_active":"2023-12-16T21:13:36.504451+0000","last_became_peered":"2023-12-16T21:13:36.504451+0000","last_unstale":"2023-12-16T21:13:36.505879+0000","last_undegraded":"2023-12-16T21:13:36.505879+0000","last_fullsized":"2023-12-16T21:13:36.505879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:06:58.471822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485417+0000","last_change":"2023-12-16T21:13:36.497205+0000","last_active":"2023-12-16T21:13:37.485417+0000","last_peered":"2023-12-16T21:13:37.485417+0000","last_clean":"2023-12-16T21:13:37.485417+0000","last_became_active":"2023-12-16T21:13:36.496919+0000","last_became_peered":"2023-12-16T21:13:36.496919+0000","last_unstale":"2023-12-16T21:13:37.485417+0000","last_undegraded":"2023-12-16T21:13:37.485417+0000","last_fullsized":"2023-12-16T21:13:37.485417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.511775+0000","last_change":"2023-12-16T21:13:36.503295+0000","last_active":"2023-12-16T21:13:36.511775+0000","last_peered":"2023-12-16T21:13:36.511775+0000","last_clean":"2023-12-16T21:13:36.511775+0000","last_became_active":"2023-12-16T21:13:36.502730+0000","last_became_peered":"2023-12-16T21:13:36.502730+0000","last_unstale":"2023-12-16T21:13:36.511775+0000","last_undegraded":"2023-12-16T21:13:36.511775+0000","last_fullsized":"2023-12-16T21:13:36.511775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:06:58.471822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485372+0000","last_change":"2023-12-16T21:13:35.508997+0000","last_active":"2023-12-16T21:13:37.485372+0000","last_peered":"2023-12-16T21:13:37.485372+0000","last_clean":"2023-12-16T21:13:37.485372+0000","last_became_active":"2023-12-16T21:13:35.508689+0000","last_became_peered":"2023-12-16T21:13:35.508689+0000","last_unstale":"2023-12-16T21:13:37.485372+0000","last_undegraded":"2023-12-16T21:13:37.485372+0000","last_fullsized":"2023-12-16T21:13:37.485372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.504701+0000","last_change":"2023-12-16T21:13:36.504100+0000","last_active":"2023-12-16T21:13:36.504701+0000","last_peered":"2023-12-16T21:13:36.504701+0000","last_clean":"2023-12-16T21:13:36.504701+0000","last_became_active":"2023-12-16T21:13:36.503340+0000","last_became_peered":"2023-12-16T21:13:36.503340+0000","last_unstale":"2023-12-16T21:13:36.504701+0000","last_undegraded":"2023-12-16T21:13:36.504701+0000","last_fullsized":"2023-12-16T21:13:36.504701+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485542+0000","last_change":"2023-12-16T21:13:35.508135+0000","last_active":"2023-12-16T21:13:37.485542+0000","last_peered":"2023-12-16T21:13:37.485542+0000","last_clean":"2023-12-16T21:13:37.485542+0000","last_became_active":"2023-12-16T21:13:35.507149+0000","last_became_peered":"2023-12-16T21:13:35.507149+0000","last_unstale":"2023-12-16T21:13:37.485542+0000","last_undegraded":"2023-12-16T21:13:37.485542+0000","last_fullsized":"2023-12-16T21:13:37.485542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485144+0000","last_change":"2023-12-16T21:13:35.510923+0000","last_active":"2023-12-16T21:13:37.485144+0000","last_peered":"2023-12-16T21:13:37.485144+0000","last_clean":"2023-12-16T21:13:37.485144+0000","last_became_active":"2023-12-16T21:13:35.510638+0000","last_became_peered":"2023-12-16T21:13:35.510638+0000","last_unstale":"2023-12-16T21:13:37.485144+0000","last_undegraded":"2023-12-16T21:13:37.485144+0000","last_fullsized":"2023-12-16T21:13:37.485144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:52.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485039+0000","last_change":"2023-12-16T21:13:36.490652+0000","last_active":"2023-12-16T21:13:37.485039+0000","last_peered":"2023-12-16T21:13:37.485039+0000","last_clean":"2023-12-16T21:13:37.485039+0000","last_became_active":"2023-12-16T21:13:36.489990+0000","last_became_peered":"2023-12-16T21:13:36.489990+0000","last_unstale":"2023-12-16T21:13:37.485039+0000","last_undegraded":"2023-12-16T21:13:37.485039+0000","last_fullsized":"2023-12-16T21:13:37.485039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486149+0000","last_change":"2023-12-16T21:13:35.498392+0000","last_active":"2023-12-16T21:13:37.486149+0000","last_peered":"2023-12-16T21:13:37.486149+0000","last_clean":"2023-12-16T21:13:37.486149+0000","last_became_active":"2023-12-16T21:13:35.498119+0000","last_became_peered":"2023-12-16T21:13:35.498119+0000","last_unstale":"2023-12-16T21:13:37.486149+0000","last_undegraded":"2023-12-16T21:13:37.486149+0000","last_fullsized":"2023-12-16T21:13:37.486149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.500917+0000","last_change":"2023-12-16T21:13:36.499009+0000","last_active":"2023-12-16T21:13:36.500917+0000","last_peered":"2023-12-16T21:13:36.500917+0000","last_clean":"2023-12-16T21:13:36.500917+0000","last_became_active":"2023-12-16T21:13:36.498714+0000","last_became_peered":"2023-12-16T21:13:36.498714+0000","last_unstale":"2023-12-16T21:13:36.500917+0000","last_undegraded":"2023-12-16T21:13:36.500917+0000","last_fullsized":"2023-12-16T21:13:36.500917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:26:23.745820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485912+0000","last_change":"2023-12-16T21:13:35.508895+0000","last_active":"2023-12-16T21:13:37.485912+0000","last_peered":"2023-12-16T21:13:37.485912+0000","last_clean":"2023-12-16T21:13:37.485912+0000","last_became_active":"2023-12-16T21:13:35.508644+0000","last_became_peered":"2023-12-16T21:13:35.508644+0000","last_unstale":"2023-12-16T21:13:37.485912+0000","last_undegraded":"2023-12-16T21:13:37.485912+0000","last_fullsized":"2023-12-16T21:13:37.485912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486627+0000","last_change":"2023-12-16T21:13:36.508270+0000","last_active":"2023-12-16T21:13:37.486627+0000","last_peered":"2023-12-16T21:13:37.486627+0000","last_clean":"2023-12-16T21:13:37.486627+0000","last_became_active":"2023-12-16T21:13:36.507998+0000","last_became_peered":"2023-12-16T21:13:36.507998+0000","last_unstale":"2023-12-16T21:13:37.486627+0000","last_undegraded":"2023-12-16T21:13:37.486627+0000","last_fullsized":"2023-12-16T21:13:37.486627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486639+0000","last_change":"2023-12-16T21:13:35.508588+0000","last_active":"2023-12-16T21:13:37.486639+0000","last_peered":"2023-12-16T21:13:37.486639+0000","last_clean":"2023-12-16T21:13:37.486639+0000","last_became_active":"2023-12-16T21:13:35.508246+0000","last_became_peered":"2023-12-16T21:13:35.508246+0000","last_unstale":"2023-12-16T21:13:37.486639+0000","last_undegraded":"2023-12-16T21:13:37.486639+0000","last_fullsized":"2023-12-16T21:13:37.486639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.508173+0000","last_change":"2023-12-16T21:13:36.507570+0000","last_active":"2023-12-16T21:13:36.508173+0000","last_peered":"2023-12-16T21:13:36.508173+0000","last_clean":"2023-12-16T21:13:36.508173+0000","last_became_active":"2023-12-16T21:13:36.507226+0000","last_became_peered":"2023-12-16T21:13:36.507226+0000","last_unstale":"2023-12-16T21:13:36.508173+0000","last_undegraded":"2023-12-16T21:13:36.508173+0000","last_fullsized":"2023-12-16T21:13:36.508173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.526963+0000","last_change":"2023-12-16T21:13:35.509646+0000","last_active":"2023-12-16T21:13:36.526963+0000","last_peered":"2023-12-16T21:13:36.526963+0000","last_clean":"2023-12-16T21:13:36.526963+0000","last_became_active":"2023-12-16T21:13:35.509335+0000","last_became_peered":"2023-12-16T21:13:35.509335+0000","last_unstale":"2023-12-16T21:13:36.526963+0000","last_undegraded":"2023-12-16T21:13:36.526963+0000","last_fullsized":"2023-12-16T21:13:36.526963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.509063+0000","last_change":"2023-12-16T21:13:36.499896+0000","last_active":"2023-12-16T21:13:36.509063+0000","last_peered":"2023-12-16T21:13:36.509063+0000","last_clean":"2023-12-16T21:13:36.509063+0000","last_became_active":"2023-12-16T21:13:36.499625+0000","last_became_peered":"2023-12-16T21:13:36.499625+0000","last_unstale":"2023-12-16T21:13:36.509063+0000","last_undegraded":"2023-12-16T21:13:36.509063+0000","last_fullsized":"2023-12-16T21:13:36.509063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:10:06.023454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486051+0000","last_change":"2023-12-16T21:13:35.507588+0000","last_active":"2023-12-16T21:13:37.486051+0000","last_peered":"2023-12-16T21:13:37.486051+0000","last_clean":"2023-12-16T21:13:37.486051+0000","last_became_active":"2023-12-16T21:13:35.507264+0000","last_became_peered":"2023-12-16T21:13:35.507264+0000","last_unstale":"2023-12-16T21:13:37.486051+0000","last_undegraded":"2023-12-16T21:13:37.486051+0000","last_fullsized":"2023-12-16T21:13:37.486051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580181+0000","last_change":"2023-12-16T21:13:36.495697+0000","last_active":"2023-12-16T21:13:37.580181+0000","last_peered":"2023-12-16T21:13:37.580181+0000","last_clean":"2023-12-16T21:13:37.580181+0000","last_became_active":"2023-12-16T21:13:36.495099+0000","last_became_peered":"2023-12-16T21:13:36.495099+0000","last_unstale":"2023-12-16T21:13:37.580181+0000","last_undegraded":"2023-12-16T21:13:37.580181+0000","last_fullsized":"2023-12-16T21:13:37.580181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.529125+0000","last_change":"2023-12-16T21:13:35.502116+0000","last_active":"2023-12-16T21:13:36.529125+0000","last_peered":"2023-12-16T21:13:36.529125+0000","last_clean":"2023-12-16T21:13:36.529125+0000","last_became_active":"2023-12-16T21:13:35.501814+0000","last_became_peered":"2023-12-16T21:13:35.501814+0000","last_unstale":"2023-12-16T21:13:36.529125+0000","last_undegraded":"2023-12-16T21:13:36.529125+0000","last_fullsized":"2023-12-16T21:13:36.529125+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501440+0000","last_change":"2023-12-16T21:13:36.499212+0000","last_active":"2023-12-16T21:13:36.501440+0000","last_peered":"2023-12-16T21:13:36.501440+0000","last_clean":"2023-12-16T21:13:36.501440+0000","last_became_active":"2023-12-16T21:13:36.498745+0000","last_became_peered":"2023-12-16T21:13:36.498745+0000","last_unstale":"2023-12-16T21:13:36.501440+0000","last_undegraded":"2023-12-16T21:13:36.501440+0000","last_fullsized":"2023-12-16T21:13:36.501440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:26.560123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486589+0000","last_change":"2023-12-16T21:13:35.506529+0000","last_active":"2023-12-16T21:13:37.486589+0000","last_peered":"2023-12-16T21:13:37.486589+0000","last_clean":"2023-12-16T21:13:37.486589+0000","last_became_active":"2023-12-16T21:13:35.506231+0000","last_became_peered":"2023-12-16T21:13:35.506231+0000","last_unstale":"2023-12-16T21:13:37.486589+0000","last_undegraded":"2023-12-16T21:13:37.486589+0000","last_fullsized":"2023-12-16T21:13:37.486589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580108+0000","last_change":"2023-12-16T21:13:36.504072+0000","last_active":"2023-12-16T21:13:37.580108+0000","last_peered":"2023-12-16T21:13:37.580108+0000","last_clean":"2023-12-16T21:13:37.580108+0000","last_became_active":"2023-12-16T21:13:36.503801+0000","last_became_peered":"2023-12-16T21:13:36.503801+0000","last_unstale":"2023-12-16T21:13:37.580108+0000","last_undegraded":"2023-12-16T21:13:37.580108+0000","last_fullsized":"2023-12-16T21:13:37.580108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485664+0000","last_change":"2023-12-16T21:13:35.508204+0000","last_active":"2023-12-16T21:13:37.485664+0000","last_peered":"2023-12-16T21:13:37.485664+0000","last_clean":"2023-12-16T21:13:37.485664+0000","last_became_active":"2023-12-16T21:13:35.507929+0000","last_became_peered":"2023-12-16T21:13:35.507929+0000","last_unstale":"2023-12-16T21:13:37.485664+0000","last_undegraded":"2023-12-16T21:13:37.485664+0000","last_fullsized":"2023-12-16T21:13:37.485664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485428+0000","last_change":"2023-12-16T21:13:36.497155+0000","last_active":"2023-12-16T21:13:37.485428+0000","last_peered":"2023-12-16T21:13:37.485428+0000","last_clean":"2023-12-16T21:13:37.485428+0000","last_became_active":"2023-12-16T21:13:36.496862+0000","last_became_peered":"2023-12-16T21:13:36.496862+0000","last_unstale":"2023-12-16T21:13:37.485428+0000","last_undegraded":"2023-12-16T21:13:37.485428+0000","last_fullsized":"2023-12-16T21:13:37.485428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.525464+0000","last_change":"2023-12-16T21:13:35.508118+0000","last_active":"2023-12-16T21:13:36.525464+0000","last_peered":"2023-12-16T21:13:36.525464+0000","last_clean":"2023-12-16T21:13:36.525464+0000","last_became_active":"2023-12-16T21:13:35.507827+0000","last_became_peered":"2023-12-16T21:13:35.507827+0000","last_unstale":"2023-12-16T21:13:36.525464+0000","last_undegraded":"2023-12-16T21:13:36.525464+0000","last_fullsized":"2023-12-16T21:13:36.525464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580055+0000","last_change":"2023-12-16T21:13:36.504720+0000","last_active":"2023-12-16T21:13:37.580055+0000","last_peered":"2023-12-16T21:13:37.580055+0000","last_clean":"2023-12-16T21:13:37.580055+0000","last_became_active":"2023-12-16T21:13:36.504444+0000","last_became_peered":"2023-12-16T21:13:36.504444+0000","last_unstale":"2023-12-16T21:13:37.580055+0000","last_undegraded":"2023-12-16T21:13:37.580055+0000","last_fullsized":"2023-12-16T21:13:37.580055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.524253+0000","last_change":"2023-12-16T21:13:35.509590+0000","last_active":"2023-12-16T21:13:36.524253+0000","last_peered":"2023-12-16T21:13:36.524253+0000","last_clean":"2023-12-16T21:13:36.524253+0000","last_became_active":"2023-12-16T21:13:35.509278+0000","last_became_peered":"2023-12-16T21:13:35.509278+0000","last_unstale":"2023-12-16T21:13:36.524253+0000","last_undegraded":"2023-12-16T21:13:36.524253+0000","last_fullsized":"2023-12-16T21:13:36.524253+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:46.659974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580433+0000","last_change":"2023-12-16T21:13:36.498151+0000","last_active":"2023-12-16T21:13:37.580433+0000","last_peered":"2023-12-16T21:13:37.580433+0000","last_clean":"2023-12-16T21:13:37.580433+0000","last_became_active":"2023-12-16T21:13:36.497907+0000","last_became_peered":"2023-12-16T21:13:36.497907+0000","last_unstale":"2023-12-16T21:13:37.580433+0000","last_undegraded":"2023-12-16T21:13:37.580433+0000","last_fullsized":"2023-12-16T21:13:37.580433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.523929+0000","last_change":"2023-12-16T21:13:35.507707+0000","last_active":"2023-12-16T21:13:36.523929+0000","last_peered":"2023-12-16T21:13:36.523929+0000","last_clean":"2023-12-16T21:13:36.523929+0000","last_became_active":"2023-12-16T21:13:35.507148+0000","last_became_peered":"2023-12-16T21:13:35.507148+0000","last_unstale":"2023-12-16T21:13:36.523929+0000","last_undegraded":"2023-12-16T21:13:36.523929+0000","last_fullsized":"2023-12-16T21:13:36.523929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580539+0000","last_change":"2023-12-16T21:13:36.503790+0000","last_active":"2023-12-16T21:13:37.580539+0000","last_peered":"2023-12-16T21:13:37.580539+0000","last_clean":"2023-12-16T21:13:37.580539+0000","last_became_active":"2023-12-16T21:13:36.503514+0000","last_became_peered":"2023-12-16T21:13:36.503514+0000","last_unstale":"2023-12-16T21:13:37.580539+0000","last_undegraded":"2023-12-16T21:13:37.580539+0000","last_fullsized":"2023-12-16T21:13:37.580539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:43:28.130579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.529137+0000","last_change":"2023-12-16T21:13:35.507575+0000","last_active":"2023-12-16T21:13:36.529137+0000","last_peered":"2023-12-16T21:13:36.529137+0000","last_clean":"2023-12-16T21:13:36.529137+0000","last_became_active":"2023-12-16T21:13:35.507291+0000","last_became_peered":"2023-12-16T21:13:35.507291+0000","last_unstale":"2023-12-16T21:13:36.529137+0000","last_undegraded":"2023-12-16T21:13:36.529137+0000","last_fullsized":"2023-12-16T21:13:36.529137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487138+0000","last_change":"2023-12-16T21:13:36.507654+0000","last_active":"2023-12-16T21:13:37.487138+0000","last_peered":"2023-12-16T21:13:37.487138+0000","last_clean":"2023-12-16T21:13:37.487138+0000","last_became_active":"2023-12-16T21:13:36.507245+0000","last_became_peered":"2023-12-16T21:13:36.507245+0000","last_unstale":"2023-12-16T21:13:37.487138+0000","last_undegraded":"2023-12-16T21:13:37.487138+0000","last_fullsized":"2023-12-16T21:13:37.487138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.487608+0000","last_change":"2023-12-16T21:13:35.509065+0000","last_active":"2023-12-16T21:13:36.487608+0000","last_peered":"2023-12-16T21:13:36.487608+0000","last_clean":"2023-12-16T21:13:36.487608+0000","last_became_active":"2023-12-16T21:13:35.508808+0000","last_became_peered":"2023-12-16T21:13:35.508808+0000","last_unstale":"2023-12-16T21:13:36.487608+0000","last_undegraded":"2023-12-16T21:13:36.487608+0000","last_fullsized":"2023-12-16T21:13:36.487608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580284+0000","last_change":"2023-12-16T21:13:36.507427+0000","last_active":"2023-12-16T21:13:37.580284+0000","last_peered":"2023-12-16T21:13:37.580284+0000","last_clean":"2023-12-16T21:13:37.580284+0000","last_became_active":"2023-12-16T21:13:36.507170+0000","last_became_peered":"2023-12-16T21:13:36.507170+0000","last_unstale":"2023-12-16T21:13:37.580284+0000","last_undegraded":"2023-12-16T21:13:37.580284+0000","last_fullsized":"2023-12-16T21:13:37.580284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490630+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:36.490630+0000","last_peered":"2023-12-16T21:13:36.490630+0000","last_clean":"2023-12-16T21:13:36.490630+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:36.490630+0000","last_undegraded":"2023-12-16T21:13:36.490630+0000","last_fullsized":"2023-12-16T21:13:36.490630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486371+0000","last_change":"2023-12-16T21:13:35.505945+0000","last_active":"2023-12-16T21:13:37.486371+0000","last_peered":"2023-12-16T21:13:37.486371+0000","last_clean":"2023-12-16T21:13:37.486371+0000","last_became_active":"2023-12-16T21:13:35.505670+0000","last_became_peered":"2023-12-16T21:13:35.505670+0000","last_unstale":"2023-12-16T21:13:37.486371+0000","last_undegraded":"2023-12-16T21:13:37.486371+0000","last_fullsized":"2023-12-16T21:13:37.486371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485341+0000","last_change":"2023-12-16T21:13:36.507596+0000","last_active":"2023-12-16T21:13:37.485341+0000","last_peered":"2023-12-16T21:13:37.485341+0000","last_clean":"2023-12-16T21:13:37.485341+0000","last_became_active":"2023-12-16T21:13:36.507302+0000","last_became_peered":"2023-12-16T21:13:36.507302+0000","last_unstale":"2023-12-16T21:13:37.485341+0000","last_undegraded":"2023-12-16T21:13:37.485341+0000","last_fullsized":"2023-12-16T21:13:37.485341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579838+0000","last_change":"2023-12-16T21:13:35.512856+0000","last_active":"2023-12-16T21:13:37.579838+0000","last_peered":"2023-12-16T21:13:37.579838+0000","last_clean":"2023-12-16T21:13:37.579838+0000","last_became_active":"2023-12-16T21:13:35.512539+0000","last_became_peered":"2023-12-16T21:13:35.512539+0000","last_unstale":"2023-12-16T21:13:37.579838+0000","last_undegraded":"2023-12-16T21:13:37.579838+0000","last_fullsized":"2023-12-16T21:13:37.579838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.504659+0000","last_change":"2023-12-16T21:13:36.503724+0000","last_active":"2023-12-16T21:13:36.504659+0000","last_peered":"2023-12-16T21:13:36.504659+0000","last_clean":"2023-12-16T21:13:36.504659+0000","last_became_active":"2023-12-16T21:13:36.503392+0000","last_became_peered":"2023-12-16T21:13:36.503392+0000","last_unstale":"2023-12-16T21:13:36.504659+0000","last_undegraded":"2023-12-16T21:13:36.504659+0000","last_fullsized":"2023-12-16T21:13:36.504659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:31:34.903149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.487011+0000","last_change":"2023-12-16T21:13:35.506777+0000","last_active":"2023-12-16T21:13:36.487011+0000","last_peered":"2023-12-16T21:13:36.487011+0000","last_clean":"2023-12-16T21:13:36.487011+0000","last_became_active":"2023-12-16T21:13:35.506483+0000","last_became_peered":"2023-12-16T21:13:35.506483+0000","last_unstale":"2023-12-16T21:13:36.487011+0000","last_undegraded":"2023-12-16T21:13:36.487011+0000","last_fullsized":"2023-12-16T21:13:36.487011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:25.556208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.497331+0000","last_change":"2023-12-16T21:13:36.496004+0000","last_active":"2023-12-16T21:13:36.497331+0000","last_peered":"2023-12-16T21:13:36.497331+0000","last_clean":"2023-12-16T21:13:36.497331+0000","last_became_active":"2023-12-16T21:13:36.495712+0000","last_became_peered":"2023-12-16T21:13:36.495712+0000","last_unstale":"2023-12-16T21:13:36.497331+0000","last_undegraded":"2023-12-16T21:13:36.497331+0000","last_fullsized":"2023-12-16T21:13:36.497331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486820+0000","last_change":"2023-12-16T21:13:35.507557+0000","last_active":"2023-12-16T21:13:37.486820+0000","last_peered":"2023-12-16T21:13:37.486820+0000","last_clean":"2023-12-16T21:13:37.486820+0000","last_became_active":"2023-12-16T21:13:35.507175+0000","last_became_peered":"2023-12-16T21:13:35.507175+0000","last_unstale":"2023-12-16T21:13:37.486820+0000","last_undegraded":"2023-12-16T21:13:37.486820+0000","last_fullsized":"2023-12-16T21:13:37.486820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:11.664911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.511648+0000","last_change":"2023-12-16T21:13:36.502502+0000","last_active":"2023-12-16T21:13:36.511648+0000","last_peered":"2023-12-16T21:13:36.511648+0000","last_clean":"2023-12-16T21:13:36.511648+0000","last_became_active":"2023-12-16T21:13:36.501893+0000","last_became_peered":"2023-12-16T21:13:36.501893+0000","last_unstale":"2023-12-16T21:13:36.511648+0000","last_undegraded":"2023-12-16T21:13:36.511648+0000","last_fullsized":"2023-12-16T21:13:36.511648+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:53.131591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490728+0000","last_change":"2023-12-16T21:13:35.513556+0000","last_active":"2023-12-16T21:13:36.490728+0000","last_peered":"2023-12-16T21:13:36.490728+0000","last_clean":"2023-12-16T21:13:36.490728+0000","last_became_active":"2023-12-16T21:13:35.513208+0000","last_became_peered":"2023-12-16T21:13:35.513208+0000","last_unstale":"2023-12-16T21:13:36.490728+0000","last_undegraded":"2023-12-16T21:13:36.490728+0000","last_fullsized":"2023-12-16T21:13:36.490728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486956+0000","last_change":"2023-12-16T21:13:36.504553+0000","last_active":"2023-12-16T21:13:37.486956+0000","last_peered":"2023-12-16T21:13:37.486956+0000","last_clean":"2023-12-16T21:13:37.486956+0000","last_became_active":"2023-12-16T21:13:36.504276+0000","last_became_peered":"2023-12-16T21:13:36.504276+0000","last_unstale":"2023-12-16T21:13:37.486956+0000","last_undegraded":"2023-12-16T21:13:37.486956+0000","last_fullsized":"2023-12-16T21:13:37.486956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580388+0000","last_change":"2023-12-16T21:13:35.511628+0000","last_active":"2023-12-16T21:13:37.580388+0000","last_peered":"2023-12-16T21:13:37.580388+0000","last_clean":"2023-12-16T21:13:37.580388+0000","last_became_active":"2023-12-16T21:13:35.510934+0000","last_became_peered":"2023-12-16T21:13:35.510934+0000","last_unstale":"2023-12-16T21:13:37.580388+0000","last_undegraded":"2023-12-16T21:13:37.580388+0000","last_fullsized":"2023-12-16T21:13:37.580388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:21.664872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.497338+0000","last_change":"2023-12-16T21:13:36.495330+0000","last_active":"2023-12-16T21:13:36.497338+0000","last_peered":"2023-12-16T21:13:36.497338+0000","last_clean":"2023-12-16T21:13:36.497338+0000","last_became_active":"2023-12-16T21:13:36.495064+0000","last_became_peered":"2023-12-16T21:13:36.495064+0000","last_unstale":"2023-12-16T21:13:36.497338+0000","last_undegraded":"2023-12-16T21:13:36.497338+0000","last_fullsized":"2023-12-16T21:13:36.497338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.493340+0000","last_change":"2023-12-16T21:13:35.508842+0000","last_active":"2023-12-16T21:13:36.493340+0000","last_peered":"2023-12-16T21:13:36.493340+0000","last_clean":"2023-12-16T21:13:36.493340+0000","last_became_active":"2023-12-16T21:13:35.508552+0000","last_became_peered":"2023-12-16T21:13:35.508552+0000","last_unstale":"2023-12-16T21:13:36.493340+0000","last_undegraded":"2023-12-16T21:13:36.493340+0000","last_fullsized":"2023-12-16T21:13:36.493340+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.497136+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.496859+0000","last_became_peered":"2023-12-16T21:13:36.496859+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486007+0000","last_change":"2023-12-16T21:13:35.512677+0000","last_active":"2023-12-16T21:13:37.486007+0000","last_peered":"2023-12-16T21:13:37.486007+0000","last_clean":"2023-12-16T21:13:37.486007+0000","last_became_active":"2023-12-16T21:13:35.511936+0000","last_became_peered":"2023-12-16T21:13:35.511936+0000","last_unstale":"2023-12-16T21:13:37.486007+0000","last_undegraded":"2023-12-16T21:13:37.486007+0000","last_fullsized":"2023-12-16T21:13:37.486007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.501183+0000","last_change":"2023-12-16T21:13:36.499822+0000","last_active":"2023-12-16T21:13:36.501183+0000","last_peered":"2023-12-16T21:13:36.501183+0000","last_clean":"2023-12-16T21:13:36.501183+0000","last_became_active":"2023-12-16T21:13:36.499517+0000","last_became_peered":"2023-12-16T21:13:36.499517+0000","last_unstale":"2023-12-16T21:13:36.501183+0000","last_undegraded":"2023-12-16T21:13:36.501183+0000","last_fullsized":"2023-12-16T21:13:36.501183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:47.663889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579837+0000","last_change":"2023-12-16T21:13:35.511783+0000","last_active":"2023-12-16T21:13:37.579837+0000","last_peered":"2023-12-16T21:13:37.579837+0000","last_clean":"2023-12-16T21:13:37.579837+0000","last_became_active":"2023-12-16T21:13:35.510752+0000","last_became_peered":"2023-12-16T21:13:35.510752+0000","last_unstale":"2023-12-16T21:13:37.579837+0000","last_undegraded":"2023-12-16T21:13:37.579837+0000","last_fullsized":"2023-12-16T21:13:37.579837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.511735+0000","last_change":"2023-12-16T21:13:36.503029+0000","last_active":"2023-12-16T21:13:36.511735+0000","last_peered":"2023-12-16T21:13:36.511735+0000","last_clean":"2023-12-16T21:13:36.511735+0000","last_became_active":"2023-12-16T21:13:36.502641+0000","last_became_peered":"2023-12-16T21:13:36.502641+0000","last_unstale":"2023-12-16T21:13:36.511735+0000","last_undegraded":"2023-12-16T21:13:36.511735+0000","last_fullsized":"2023-12-16T21:13:36.511735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:26.560123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.494285+0000","last_change":"2023-12-16T21:13:35.511492+0000","last_active":"2023-12-16T21:13:36.494285+0000","last_peered":"2023-12-16T21:13:36.494285+0000","last_clean":"2023-12-16T21:13:36.494285+0000","last_became_active":"2023-12-16T21:13:35.511118+0000","last_became_peered":"2023-12-16T21:13:35.511118+0000","last_unstale":"2023-12-16T21:13:36.494285+0000","last_undegraded":"2023-12-16T21:13:36.494285+0000","last_fullsized":"2023-12-16T21:13:36.494285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485112+0000","last_change":"2023-12-16T21:13:36.495196+0000","last_active":"2023-12-16T21:13:37.485112+0000","last_peered":"2023-12-16T21:13:37.485112+0000","last_clean":"2023-12-16T21:13:37.485112+0000","last_became_active":"2023-12-16T21:13:36.494933+0000","last_became_peered":"2023-12-16T21:13:36.494933+0000","last_unstale":"2023-12-16T21:13:37.485112+0000","last_undegraded":"2023-12-16T21:13:37.485112+0000","last_fullsized":"2023-12-16T21:13:37.485112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.494120+0000","last_change":"2023-12-16T21:13:35.506652+0000","last_active":"2023-12-16T21:13:36.494120+0000","last_peered":"2023-12-16T21:13:36.494120+0000","last_clean":"2023-12-16T21:13:36.494120+0000","last_became_active":"2023-12-16T21:13:35.506349+0000","last_became_peered":"2023-12-16T21:13:35.506349+0000","last_unstale":"2023-12-16T21:13:36.494120+0000","last_undegraded":"2023-12-16T21:13:36.494120+0000","last_fullsized":"2023-12-16T21:13:36.494120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485338+0000","last_change":"2023-12-16T21:13:36.490929+0000","last_active":"2023-12-16T21:13:37.485338+0000","last_peered":"2023-12-16T21:13:37.485338+0000","last_clean":"2023-12-16T21:13:37.485338+0000","last_became_active":"2023-12-16T21:13:36.490557+0000","last_became_peered":"2023-12-16T21:13:36.490557+0000","last_unstale":"2023-12-16T21:13:37.485338+0000","last_undegraded":"2023-12-16T21:13:37.485338+0000","last_fullsized":"2023-12-16T21:13:37.485338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580200+0000","last_change":"2023-12-16T21:13:35.511062+0000","last_active":"2023-12-16T21:13:37.580200+0000","last_peered":"2023-12-16T21:13:37.580200+0000","last_clean":"2023-12-16T21:13:37.580200+0000","last_became_active":"2023-12-16T21:13:35.510780+0000","last_became_peered":"2023-12-16T21:13:35.510780+0000","last_unstale":"2023-12-16T21:13:37.580200+0000","last_undegraded":"2023-12-16T21:13:37.580200+0000","last_fullsized":"2023-12-16T21:13:37.580200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:25.707555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485439+0000","last_change":"2023-12-16T21:13:36.499546+0000","last_active":"2023-12-16T21:13:37.485439+0000","last_peered":"2023-12-16T21:13:37.485439+0000","last_clean":"2023-12-16T21:13:37.485439+0000","last_became_active":"2023-12-16T21:13:36.499236+0000","last_became_peered":"2023-12-16T21:13:36.499236+0000","last_unstale":"2023-12-16T21:13:37.485439+0000","last_undegraded":"2023-12-16T21:13:37.485439+0000","last_fullsized":"2023-12-16T21:13:37.485439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486745+0000","last_change":"2023-12-16T21:13:36.512546+0000","last_active":"2023-12-16T21:13:37.486745+0000","last_peered":"2023-12-16T21:13:37.486745+0000","last_clean":"2023-12-16T21:13:37.486745+0000","last_became_active":"2023-12-16T21:13:36.512196+0000","last_became_peered":"2023-12-16T21:13:36.512196+0000","last_unstale":"2023-12-16T21:13:37.486745+0000","last_undegraded":"2023-12-16T21:13:37.486745+0000","last_fullsized":"2023-12-16T21:13:37.486745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580134+0000","last_change":"2023-12-16T21:13:35.511999+0000","last_active":"2023-12-16T21:13:37.580134+0000","last_peered":"2023-12-16T21:13:37.580134+0000","last_clean":"2023-12-16T21:13:37.580134+0000","last_became_active":"2023-12-16T21:13:35.511760+0000","last_became_peered":"2023-12-16T21:13:35.511760+0000","last_unstale":"2023-12-16T21:13:37.580134+0000","last_undegraded":"2023-12-16T21:13:37.580134+0000","last_fullsized":"2023-12-16T21:13:37.580134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:49.269497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487061+0000","last_change":"2023-12-16T21:13:36.512673+0000","last_active":"2023-12-16T21:13:37.487061+0000","last_peered":"2023-12-16T21:13:37.487061+0000","last_clean":"2023-12-16T21:13:37.487061+0000","last_became_active":"2023-12-16T21:13:36.512185+0000","last_became_peered":"2023-12-16T21:13:36.512185+0000","last_unstale":"2023-12-16T21:13:37.487061+0000","last_undegraded":"2023-12-16T21:13:37.487061+0000","last_fullsized":"2023-12-16T21:13:37.487061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580255+0000","last_change":"2023-12-16T21:13:35.511982+0000","last_active":"2023-12-16T21:13:37.580255+0000","last_peered":"2023-12-16T21:13:37.580255+0000","last_clean":"2023-12-16T21:13:37.580255+0000","last_became_active":"2023-12-16T21:13:35.511359+0000","last_became_peered":"2023-12-16T21:13:35.511359+0000","last_unstale":"2023-12-16T21:13:37.580255+0000","last_undegraded":"2023-12-16T21:13:37.580255+0000","last_fullsized":"2023-12-16T21:13:37.580255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.506611+0000","last_change":"2023-12-16T21:13:36.505890+0000","last_active":"2023-12-16T21:13:36.506611+0000","last_peered":"2023-12-16T21:13:36.506611+0000","last_clean":"2023-12-16T21:13:36.506611+0000","last_became_active":"2023-12-16T21:13:36.505414+0000","last_became_peered":"2023-12-16T21:13:36.505414+0000","last_unstale":"2023-12-16T21:13:36.506611+0000","last_undegraded":"2023-12-16T21:13:36.506611+0000","last_fullsized":"2023-12-16T21:13:36.506611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485644+0000","last_change":"2023-12-16T21:13:35.511468+0000","last_active":"2023-12-16T21:13:37.485644+0000","last_peered":"2023-12-16T21:13:37.485644+0000","last_clean":"2023-12-16T21:13:37.485644+0000","last_became_active":"2023-12-16T21:13:35.511092+0000","last_became_peered":"2023-12-16T21:13:35.511092+0000","last_unstale":"2023-12-16T21:13:37.485644+0000","last_undegraded":"2023-12-16T21:13:37.485644+0000","last_fullsized":"2023-12-16T21:13:37.485644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485618+0000","last_change":"2023-12-16T21:13:36.494416+0000","last_active":"2023-12-16T21:13:37.485618+0000","last_peered":"2023-12-16T21:13:37.485618+0000","last_clean":"2023-12-16T21:13:37.485618+0000","last_became_active":"2023-12-16T21:13:36.494121+0000","last_became_peered":"2023-12-16T21:13:36.494121+0000","last_unstale":"2023-12-16T21:13:37.485618+0000","last_undegraded":"2023-12-16T21:13:37.485618+0000","last_fullsized":"2023-12-16T21:13:37.485618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.529329+0000","last_change":"2023-12-16T21:13:35.510139+0000","last_active":"2023-12-16T21:13:36.529329+0000","last_peered":"2023-12-16T21:13:36.529329+0000","last_clean":"2023-12-16T21:13:36.529329+0000","last_became_active":"2023-12-16T21:13:35.509804+0000","last_became_peered":"2023-12-16T21:13:35.509804+0000","last_unstale":"2023-12-16T21:13:36.529329+0000","last_undegraded":"2023-12-16T21:13:36.529329+0000","last_fullsized":"2023-12-16T21:13:36.529329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:56:59.813907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":660231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":69407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:43.958 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.077+0000 7fb2c3e2d640 1 Processor -- start 2023-12-16T21:13:44.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.078+0000 7fb2c3e2d640 1 -- start start 2023-12-16T21:13:44.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.078+0000 7fb2c3e2d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c3e2d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2bc0d1d50 0x7fb2bc06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c3e2d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2bc06ce00 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c3e2d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2bc06bf40 con 0x7fb2bc0d1d50 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c3e2d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc06c1f0 con 0x7fb2bc15d6c0 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2bc0d1d50 0x7fb2bc06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb2bc0d1d50 0x7fb2bc06ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36052/0 (socket says 172.21.15.46:36052) 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 -- 172.21.15.46:0/45718421 learned_addr learned my addr 172.21.15.46:0/45718421 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c13c8640 1 --2- 172.21.15.46:0/45718421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 -- 172.21.15.46:0/45718421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 msgr2=0x7fb2bc06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 --2- 172.21.15.46:0/45718421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc06b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 -- 172.21.15.46:0/45718421 >> v1:172.21.15.121:6789/0 conn(0x7fb2bc15d2e0 legacy=0x7fb2bc0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.079+0000 7fb2c23ca640 1 -- 172.21.15.46:0/45718421 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc06c4d0 con 0x7fb2bc0d1d50 2023-12-16T21:13:44.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.080+0000 7fb2c23ca640 1 --2- 172.21.15.46:0/45718421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc06ba00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009e50 tx=0x7fb2b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93873236d3c90bc5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.080+0000 7fb2c0bc7640 1 -- 172.21.15.46:0/45718421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b402ca70 con 0x7fb2bc0d1d50 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.080+0000 7fb2c0bc7640 1 -- 172.21.15.46:0/45718421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b402cbd0 con 0x7fb2bc0d1d50 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.080+0000 7fb2c0bc7640 1 -- 172.21.15.46:0/45718421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40343f0 con 0x7fb2bc0d1d50 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/45718421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc0d1d50 msgr2=0x7fb2bc06ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/45718421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc06ba00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4009e50 tx=0x7fb2b402bed0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/45718421 shutdown_connections 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/45718421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc06b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/45718421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc06ba00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/45718421 >> 172.21.15.46:0/45718421 conn(0x7fb2bc09c580 msgr2=0x7fb2bc0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:44.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/45718421 shutdown_connections 2023-12-16T21:13:44.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.081+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/45718421 wait complete. 2023-12-16T21:13:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.082+0000 7fb2c3e2d640 1 Processor -- start 2023-12-16T21:13:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.082+0000 7fb2c3e2d640 1 -- start start 2023-12-16T21:13:44.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c3e2d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc1f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c3e2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 0x7fb2bc0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c3e2d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc1f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c3e2d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc15b560 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c3e2d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc15b7b0 con 0x7fb2bc0d1d50 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c3e2d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2bc15ba90 con 0x7fb2bc15d6c0 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 0x7fb2bc0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 0x7fb2bc0ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36062/0 (socket says 172.21.15.46:36062) 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 -- 172.21.15.46:0/970504916 learned_addr learned my addr 172.21.15.46:0/970504916 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c23ca640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc1f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c1bc9640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc1f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 -- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 msgr2=0x7fb2bc1f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc1f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 -- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2bc0d1d50 msgr2=0x7fb2bc1f6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc1f6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 -- 172.21.15.46:0/970504916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b4009b30 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.083+0000 7fb2c13c8640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 0x7fb2bc0ccd40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00d5c0 tx=0x7fb2ac00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.084+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0125f0 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.084+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ac012750 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.084+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2bc1f63c0 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.084+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac012b00 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.084+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc1f6980 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.085+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2bc1f6b40 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.085+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb2ac012c60 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.085+0000 7fb2b27fc640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb2a006ae40 0x7fb2a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.086+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2ac089ad0 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.086+0000 7fb2c1bc9640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb2a006ae40 0x7fb2a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.086+0000 7fb2c1bc9640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb2a006ae40 0x7fb2a006d300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8000f60 tx=0x7fb2b8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.090+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb2ac054b80 con 0x7fb2bc15d2e0 2023-12-16T21:13:44.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.262+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2bc15b920 con 0x7fb2a006ae40 2023-12-16T21:13:44.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.269+0000 7fb2b27fc640 1 -- 172.21.15.46:0/970504916 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264558 (secure 0 0 0) 0x7fb2bc15b920 con 0x7fb2a006ae40 2023-12-16T21:13:44.272 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:44.273 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:44.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb2a006ae40 msgr2=0x7fb2a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb2a006ae40 0x7fb2a006d300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8000f60 tx=0x7fb2b8009040 comp rx=0 tx=0).stop 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 msgr2=0x7fb2bc0ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 0x7fb2bc0ccd40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00d5c0 tx=0x7fb2ac00da80 comp rx=0 tx=0).stop 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 shutdown_connections 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb2a006ae40 0x7fb2a006d300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb2bc15d2e0 0x7fb2bc0ccd40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2bc15d6c0 0x7fb2bc1f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 --2- 172.21.15.46:0/970504916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2bc0d1d50 0x7fb2bc1f6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 >> 172.21.15.46:0/970504916 conn(0x7fb2bc09c580 msgr2=0x7fb2bc0b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 shutdown_connections 2023-12-16T21:13:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.272+0000 7fb2c3e2d640 1 -- 172.21.15.46:0/970504916 wait complete. 2023-12-16T21:13:44.282 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-16T21:13:43.956734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235752,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738968,"statfs":{"total":773094113280,"available":772852703232,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002101"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580448+0000","last_change":"2023-12-16T21:13:36.497094+0000","last_active":"2023-12-16T21:13:37.580448+0000","last_peered":"2023-12-16T21:13:37.580448+0000","last_clean":"2023-12-16T21:13:37.580448+0000","last_became_active":"2023-12-16T21:13:36.496839+0000","last_became_peered":"2023-12-16T21:13:36.496839+0000","last_unstale":"2023-12-16T21:13:37.580448+0000","last_undegraded":"2023-12-16T21:13:37.580448+0000","last_fullsized":"2023-12-16T21:13:37.580448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043043+0000","last_change":"2023-12-16T21:13:36.496641+0000","last_active":"2023-12-16T21:13:38.043043+0000","last_peered":"2023-12-16T21:13:38.043043+0000","last_clean":"2023-12-16T21:13:38.043043+0000","last_became_active":"2023-12-16T21:13:36.496301+0000","last_became_peered":"2023-12-16T21:13:36.496301+0000","last_unstale":"2023-12-16T21:13:38.043043+0000","last_undegraded":"2023-12-16T21:13:38.043043+0000","last_fullsized":"2023-12-16T21:13:38.043043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004757+0000","last_change":"2023-12-16T21:13:36.490436+0000","last_active":"2023-12-16T21:13:38.004757+0000","last_peered":"2023-12-16T21:13:38.004757+0000","last_clean":"2023-12-16T21:13:38.004757+0000","last_became_active":"2023-12-16T21:13:36.489881+0000","last_became_peered":"2023-12-16T21:13:36.489881+0000","last_unstale":"2023-12-16T21:13:38.004757+0000","last_undegraded":"2023-12-16T21:13:38.004757+0000","last_fullsized":"2023-12-16T21:13:38.004757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487045+0000","last_change":"2023-12-16T21:13:36.507326+0000","last_active":"2023-12-16T21:13:37.487045+0000","last_peered":"2023-12-16T21:13:37.487045+0000","last_clean":"2023-12-16T21:13:37.487045+0000","last_became_active":"2023-12-16T21:13:36.507063+0000","last_became_peered":"2023-12-16T21:13:36.507063+0000","last_unstale":"2023-12-16T21:13:37.487045+0000","last_undegraded":"2023-12-16T21:13:37.487045+0000","last_fullsized":"2023-12-16T21:13:37.487045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580048+0000","last_change":"2023-12-16T21:13:36.494596+0000","last_active":"2023-12-16T21:13:37.580048+0000","last_peered":"2023-12-16T21:13:37.580048+0000","last_clean":"2023-12-16T21:13:37.580048+0000","last_became_active":"2023-12-16T21:13:36.494238+0000","last_became_peered":"2023-12-16T21:13:36.494238+0000","last_unstale":"2023-12-16T21:13:37.580048+0000","last_undegraded":"2023-12-16T21:13:37.580048+0000","last_fullsized":"2023-12-16T21:13:37.580048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486248+0000","last_change":"2023-12-16T21:13:36.505505+0000","last_active":"2023-12-16T21:13:37.486248+0000","last_peered":"2023-12-16T21:13:37.486248+0000","last_clean":"2023-12-16T21:13:37.486248+0000","last_became_active":"2023-12-16T21:13:36.505205+0000","last_became_peered":"2023-12-16T21:13:36.505205+0000","last_unstale":"2023-12-16T21:13:37.486248+0000","last_undegraded":"2023-12-16T21:13:37.486248+0000","last_fullsized":"2023-12-16T21:13:37.486248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579992+0000","last_change":"2023-12-16T21:13:36.502009+0000","last_active":"2023-12-16T21:13:37.579992+0000","last_peered":"2023-12-16T21:13:37.579992+0000","last_clean":"2023-12-16T21:13:37.579992+0000","last_became_active":"2023-12-16T21:13:36.501672+0000","last_became_peered":"2023-12-16T21:13:36.501672+0000","last_unstale":"2023-12-16T21:13:37.579992+0000","last_undegraded":"2023-12-16T21:13:37.579992+0000","last_fullsized":"2023-12-16T21:13:37.579992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.484971+0000","last_change":"2023-12-16T21:13:36.502551+0000","last_active":"2023-12-16T21:13:37.484971+0000","last_peered":"2023-12-16T21:13:37.484971+0000","last_clean":"2023-12-16T21:13:37.484971+0000","last_became_active":"2023-12-16T21:13:36.502214+0000","last_became_peered":"2023-12-16T21:13:36.502214+0000","last_unstale":"2023-12-16T21:13:37.484971+0000","last_undegraded":"2023-12-16T21:13:37.484971+0000","last_fullsized":"2023-12-16T21:13:37.484971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580474+0000","last_change":"2023-12-16T21:13:36.507546+0000","last_active":"2023-12-16T21:13:37.580474+0000","last_peered":"2023-12-16T21:13:37.580474+0000","last_clean":"2023-12-16T21:13:37.580474+0000","last_became_active":"2023-12-16T21:13:36.507253+0000","last_became_peered":"2023-12-16T21:13:36.507253+0000","last_unstale":"2023-12-16T21:13:37.580474+0000","last_undegraded":"2023-12-16T21:13:37.580474+0000","last_fullsized":"2023-12-16T21:13:37.580474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:22:24.104289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485901+0000","last_change":"2023-12-16T21:13:36.508602+0000","last_active":"2023-12-16T21:13:37.485901+0000","last_peered":"2023-12-16T21:13:37.485901+0000","last_clean":"2023-12-16T21:13:37.485901+0000","last_became_active":"2023-12-16T21:13:36.508223+0000","last_became_peered":"2023-12-16T21:13:36.508223+0000","last_unstale":"2023-12-16T21:13:37.485901+0000","last_undegraded":"2023-12-16T21:13:37.485901+0000","last_fullsized":"2023-12-16T21:13:37.485901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485806+0000","last_change":"2023-12-16T21:13:36.508624+0000","last_active":"2023-12-16T21:13:37.485806+0000","last_peered":"2023-12-16T21:13:37.485806+0000","last_clean":"2023-12-16T21:13:37.485806+0000","last_became_active":"2023-12-16T21:13:36.508293+0000","last_became_peered":"2023-12-16T21:13:36.508293+0000","last_unstale":"2023-12-16T21:13:37.485806+0000","last_undegraded":"2023-12-16T21:13:37.485806+0000","last_fullsized":"2023-12-16T21:13:37.485806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043125+0000","last_change":"2023-12-16T21:13:36.499195+0000","last_active":"2023-12-16T21:13:38.043125+0000","last_peered":"2023-12-16T21:13:38.043125+0000","last_clean":"2023-12-16T21:13:38.043125+0000","last_became_active":"2023-12-16T21:13:36.498927+0000","last_became_peered":"2023-12-16T21:13:36.498927+0000","last_unstale":"2023-12-16T21:13:38.043125+0000","last_undegraded":"2023-12-16T21:13:38.043125+0000","last_fullsized":"2023-12-16T21:13:38.043125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485287+0000","last_change":"2023-12-16T21:13:36.494278+0000","last_active":"2023-12-16T21:13:37.485287+0000","last_peered":"2023-12-16T21:13:37.485287+0000","last_clean":"2023-12-16T21:13:37.485287+0000","last_became_active":"2023-12-16T21:13:36.493970+0000","last_became_peered":"2023-12-16T21:13:36.493970+0000","last_unstale":"2023-12-16T21:13:37.485287+0000","last_undegraded":"2023-12-16T21:13:37.485287+0000","last_fullsized":"2023-12-16T21:13:37.485287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004516+0000","last_change":"2023-12-16T21:13:36.499374+0000","last_active":"2023-12-16T21:13:38.004516+0000","last_peered":"2023-12-16T21:13:38.004516+0000","last_clean":"2023-12-16T21:13:38.004516+0000","last_became_active":"2023-12-16T21:13:36.499097+0000","last_became_peered":"2023-12-16T21:13:36.499097+0000","last_unstale":"2023-12-16T21:13:38.004516+0000","last_undegraded":"2023-12-16T21:13:38.004516+0000","last_fullsized":"2023-12-16T21:13:38.004516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:33.841187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042953+0000","last_change":"2023-12-16T21:13:36.504350+0000","last_active":"2023-12-16T21:13:38.042953+0000","last_peered":"2023-12-16T21:13:38.042953+0000","last_clean":"2023-12-16T21:13:38.042953+0000","last_became_active":"2023-12-16T21:13:36.504046+0000","last_became_peered":"2023-12-16T21:13:36.504046+0000","last_unstale":"2023-12-16T21:13:38.042953+0000","last_undegraded":"2023-12-16T21:13:38.042953+0000","last_fullsized":"2023-12-16T21:13:38.042953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485909+0000","last_change":"2023-12-16T21:13:36.502440+0000","last_active":"2023-12-16T21:13:37.485909+0000","last_peered":"2023-12-16T21:13:37.485909+0000","last_clean":"2023-12-16T21:13:37.485909+0000","last_became_active":"2023-12-16T21:13:36.502160+0000","last_became_peered":"2023-12-16T21:13:36.502160+0000","last_unstale":"2023-12-16T21:13:37.485909+0000","last_undegraded":"2023-12-16T21:13:37.485909+0000","last_fullsized":"2023-12-16T21:13:37.485909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487000+0000","last_change":"2023-12-16T21:13:36.508778+0000","last_active":"2023-12-16T21:13:37.487000+0000","last_peered":"2023-12-16T21:13:37.487000+0000","last_clean":"2023-12-16T21:13:37.487000+0000","last_became_active":"2023-12-16T21:13:36.508506+0000","last_became_peered":"2023-12-16T21:13:36.508506+0000","last_unstale":"2023-12-16T21:13:37.487000+0000","last_undegraded":"2023-12-16T21:13:37.487000+0000","last_fullsized":"2023-12-16T21:13:37.487000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042744+0000","last_change":"2023-12-16T21:13:36.494459+0000","last_active":"2023-12-16T21:13:38.042744+0000","last_peered":"2023-12-16T21:13:38.042744+0000","last_clean":"2023-12-16T21:13:38.042744+0000","last_became_active":"2023-12-16T21:13:36.494156+0000","last_became_peered":"2023-12-16T21:13:36.494156+0000","last_unstale":"2023-12-16T21:13:38.042744+0000","last_undegraded":"2023-12-16T21:13:38.042744+0000","last_fullsized":"2023-12-16T21:13:38.042744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:47.886346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580395+0000","last_change":"2023-12-16T21:13:36.496446+0000","last_active":"2023-12-16T21:13:37.580395+0000","last_peered":"2023-12-16T21:13:37.580395+0000","last_clean":"2023-12-16T21:13:37.580395+0000","last_became_active":"2023-12-16T21:13:36.496123+0000","last_became_peered":"2023-12-16T21:13:36.496123+0000","last_unstale":"2023-12-16T21:13:37.580395+0000","last_undegraded":"2023-12-16T21:13:37.580395+0000","last_fullsized":"2023-12-16T21:13:37.580395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486172+0000","last_change":"2023-12-16T21:13:36.505223+0000","last_active":"2023-12-16T21:13:37.486172+0000","last_peered":"2023-12-16T21:13:37.486172+0000","last_clean":"2023-12-16T21:13:37.486172+0000","last_became_active":"2023-12-16T21:13:36.504834+0000","last_became_peered":"2023-12-16T21:13:36.504834+0000","last_unstale":"2023-12-16T21:13:37.486172+0000","last_undegraded":"2023-12-16T21:13:37.486172+0000","last_fullsized":"2023-12-16T21:13:37.486172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004625+0000","last_change":"2023-12-16T21:13:36.500880+0000","last_active":"2023-12-16T21:13:38.004625+0000","last_peered":"2023-12-16T21:13:38.004625+0000","last_clean":"2023-12-16T21:13:38.004625+0000","last_became_active":"2023-12-16T21:13:36.500532+0000","last_became_peered":"2023-12-16T21:13:36.500532+0000","last_unstale":"2023-12-16T21:13:38.004625+0000","last_undegraded":"2023-12-16T21:13:38.004625+0000","last_fullsized":"2023-12-16T21:13:38.004625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486491+0000","last_change":"2023-12-16T21:13:36.507578+0000","last_active":"2023-12-16T21:13:37.486491+0000","last_peered":"2023-12-16T21:13:37.486491+0000","last_clean":"2023-12-16T21:13:37.486491+0000","last_became_active":"2023-12-16T21:13:36.507206+0000","last_became_peered":"2023-12-16T21:13:36.507206+0000","last_unstale":"2023-12-16T21:13:37.486491+0000","last_undegraded":"2023-12-16T21:13:37.486491+0000","last_fullsized":"2023-12-16T21:13:37.486491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485167+0000","last_change":"2023-12-16T21:13:36.506029+0000","last_active":"2023-12-16T21:13:37.485167+0000","last_peered":"2023-12-16T21:13:37.485167+0000","last_clean":"2023-12-16T21:13:37.485167+0000","last_became_active":"2023-12-16T21:13:36.505785+0000","last_became_peered":"2023-12-16T21:13:36.505785+0000","last_unstale":"2023-12-16T21:13:37.485167+0000","last_undegraded":"2023-12-16T21:13:37.485167+0000","last_fullsized":"2023-12-16T21:13:37.485167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486054+0000","last_change":"2023-12-16T21:13:36.504916+0000","last_active":"2023-12-16T21:13:37.486054+0000","last_peered":"2023-12-16T21:13:37.486054+0000","last_clean":"2023-12-16T21:13:37.486054+0000","last_became_active":"2023-12-16T21:13:36.504422+0000","last_became_peered":"2023-12-16T21:13:36.504422+0000","last_unstale":"2023-12-16T21:13:37.486054+0000","last_undegraded":"2023-12-16T21:13:37.486054+0000","last_fullsized":"2023-12-16T21:13:37.486054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.503543+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.503210+0000","last_became_peered":"2023-12-16T21:13:36.503210+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486065+0000","last_change":"2023-12-16T21:13:36.494611+0000","last_active":"2023-12-16T21:13:37.486065+0000","last_peered":"2023-12-16T21:13:37.486065+0000","last_clean":"2023-12-16T21:13:37.486065+0000","last_became_active":"2023-12-16T21:13:36.494373+0000","last_became_peered":"2023-12-16T21:13:36.494373+0000","last_unstale":"2023-12-16T21:13:37.486065+0000","last_undegraded":"2023-12-16T21:13:37.486065+0000","last_fullsized":"2023-12-16T21:13:37.486065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043012+0000","last_change":"2023-12-16T21:13:36.498640+0000","last_active":"2023-12-16T21:13:38.043012+0000","last_peered":"2023-12-16T21:13:38.043012+0000","last_clean":"2023-12-16T21:13:38.043012+0000","last_became_active":"2023-12-16T21:13:36.498334+0000","last_became_peered":"2023-12-16T21:13:36.498334+0000","last_unstale":"2023-12-16T21:13:38.043012+0000","last_undegraded":"2023-12-16T21:13:38.043012+0000","last_fullsized":"2023-12-16T21:13:38.043012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485419+0000","last_change":"2023-12-16T21:13:36.506001+0000","last_active":"2023-12-16T21:13:37.485419+0000","last_peered":"2023-12-16T21:13:37.485419+0000","last_clean":"2023-12-16T21:13:37.485419+0000","last_became_active":"2023-12-16T21:13:36.505724+0000","last_became_peered":"2023-12-16T21:13:36.505724+0000","last_unstale":"2023-12-16T21:13:37.485419+0000","last_undegraded":"2023-12-16T21:13:37.485419+0000","last_fullsized":"2023-12-16T21:13:37.485419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004718+0000","last_change":"2023-12-16T21:13:36.489974+0000","last_active":"2023-12-16T21:13:38.004718+0000","last_peered":"2023-12-16T21:13:38.004718+0000","last_clean":"2023-12-16T21:13:38.004718+0000","last_became_active":"2023-12-16T21:13:36.489597+0000","last_became_peered":"2023-12-16T21:13:36.489597+0000","last_unstale":"2023-12-16T21:13:38.004718+0000","last_undegraded":"2023-12-16T21:13:38.004718+0000","last_fullsized":"2023-12-16T21:13:38.004718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486570+0000","last_change":"2023-12-16T21:13:36.505535+0000","last_active":"2023-12-16T21:13:37.486570+0000","last_peered":"2023-12-16T21:13:37.486570+0000","last_clean":"2023-12-16T21:13:37.486570+0000","last_became_active":"2023-12-16T21:13:36.505185+0000","last_became_peered":"2023-12-16T21:13:36.505185+0000","last_unstale":"2023-12-16T21:13:37.486570+0000","last_undegraded":"2023-12-16T21:13:37.486570+0000","last_fullsized":"2023-12-16T21:13:37.486570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042731+0000","last_change":"2023-12-16T21:13:36.504923+0000","last_active":"2023-12-16T21:13:38.042731+0000","last_peered":"2023-12-16T21:13:38.042731+0000","last_clean":"2023-12-16T21:13:38.042731+0000","last_became_active":"2023-12-16T21:13:36.504451+0000","last_became_peered":"2023-12-16T21:13:36.504451+0000","last_unstale":"2023-12-16T21:13:38.042731+0000","last_undegraded":"2023-12-16T21:13:38.042731+0000","last_fullsized":"2023-12-16T21:13:38.042731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485417+0000","last_change":"2023-12-16T21:13:36.497205+0000","last_active":"2023-12-16T21:13:37.485417+0000","last_peered":"2023-12-16T21:13:37.485417+0000","last_clean":"2023-12-16T21:13:37.485417+0000","last_became_active":"2023-12-16T21:13:36.496919+0000","last_became_peered":"2023-12-16T21:13:36.496919+0000","last_unstale":"2023-12-16T21:13:37.485417+0000","last_undegraded":"2023-12-16T21:13:37.485417+0000","last_fullsized":"2023-12-16T21:13:37.485417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042855+0000","last_change":"2023-12-16T21:13:36.503295+0000","last_active":"2023-12-16T21:13:38.042855+0000","last_peered":"2023-12-16T21:13:38.042855+0000","last_clean":"2023-12-16T21:13:38.042855+0000","last_became_active":"2023-12-16T21:13:36.502730+0000","last_became_peered":"2023-12-16T21:13:36.502730+0000","last_unstale":"2023-12-16T21:13:38.042855+0000","last_undegraded":"2023-12-16T21:13:38.042855+0000","last_fullsized":"2023-12-16T21:13:38.042855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485372+0000","last_change":"2023-12-16T21:13:35.508997+0000","last_active":"2023-12-16T21:13:37.485372+0000","last_peered":"2023-12-16T21:13:37.485372+0000","last_clean":"2023-12-16T21:13:37.485372+0000","last_became_active":"2023-12-16T21:13:35.508689+0000","last_became_peered":"2023-12-16T21:13:35.508689+0000","last_unstale":"2023-12-16T21:13:37.485372+0000","last_undegraded":"2023-12-16T21:13:37.485372+0000","last_fullsized":"2023-12-16T21:13:37.485372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004783+0000","last_change":"2023-12-16T21:13:36.504100+0000","last_active":"2023-12-16T21:13:38.004783+0000","last_peered":"2023-12-16T21:13:38.004783+0000","last_clean":"2023-12-16T21:13:38.004783+0000","last_became_active":"2023-12-16T21:13:36.503340+0000","last_became_peered":"2023-12-16T21:13:36.503340+0000","last_unstale":"2023-12-16T21:13:38.004783+0000","last_undegraded":"2023-12-16T21:13:38.004783+0000","last_fullsized":"2023-12-16T21:13:38.004783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485542+0000","last_change":"2023-12-16T21:13:35.508135+0000","last_active":"2023-12-16T21:13:37.485542+0000","last_peered":"2023-12-16T21:13:37.485542+0000","last_clean":"2023-12-16T21:13:37.485542+0000","last_became_active":"2023-12-16T21:13:35.507149+0000","last_became_peered":"2023-12-16T21:13:35.507149+0000","last_unstale":"2023-12-16T21:13:37.485542+0000","last_undegraded":"2023-12-16T21:13:37.485542+0000","last_fullsized":"2023-12-16T21:13:37.485542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485144+0000","last_change":"2023-12-16T21:13:35.510923+0000","last_active":"2023-12-16T21:13:37.485144+0000","last_peered":"2023-12-16T21:13:37.485144+0000","last_clean":"2023-12-16T21:13:37.485144+0000","last_became_active":"2023-12-16T21:13:35.510638+0000","last_became_peered":"2023-12-16T21:13:35.510638+0000","last_unstale":"2023-12-16T21:13:37.485144+0000","last_undegraded":"2023-12-16T21:13:37.485144+0000","last_fullsized":"2023-12-16T21:13:37.485144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:52.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485039+0000","last_change":"2023-12-16T21:13:36.490652+0000","last_active":"2023-12-16T21:13:37.485039+0000","last_peered":"2023-12-16T21:13:37.485039+0000","last_clean":"2023-12-16T21:13:37.485039+0000","last_became_active":"2023-12-16T21:13:36.489990+0000","last_became_peered":"2023-12-16T21:13:36.489990+0000","last_unstale":"2023-12-16T21:13:37.485039+0000","last_undegraded":"2023-12-16T21:13:37.485039+0000","last_fullsized":"2023-12-16T21:13:37.485039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486149+0000","last_change":"2023-12-16T21:13:35.498392+0000","last_active":"2023-12-16T21:13:37.486149+0000","last_peered":"2023-12-16T21:13:37.486149+0000","last_clean":"2023-12-16T21:13:37.486149+0000","last_became_active":"2023-12-16T21:13:35.498119+0000","last_became_peered":"2023-12-16T21:13:35.498119+0000","last_unstale":"2023-12-16T21:13:37.486149+0000","last_undegraded":"2023-12-16T21:13:37.486149+0000","last_fullsized":"2023-12-16T21:13:37.486149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042709+0000","last_change":"2023-12-16T21:13:36.499009+0000","last_active":"2023-12-16T21:13:38.042709+0000","last_peered":"2023-12-16T21:13:38.042709+0000","last_clean":"2023-12-16T21:13:38.042709+0000","last_became_active":"2023-12-16T21:13:36.498714+0000","last_became_peered":"2023-12-16T21:13:36.498714+0000","last_unstale":"2023-12-16T21:13:38.042709+0000","last_undegraded":"2023-12-16T21:13:38.042709+0000","last_fullsized":"2023-12-16T21:13:38.042709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485912+0000","last_change":"2023-12-16T21:13:35.508895+0000","last_active":"2023-12-16T21:13:37.485912+0000","last_peered":"2023-12-16T21:13:37.485912+0000","last_clean":"2023-12-16T21:13:37.485912+0000","last_became_active":"2023-12-16T21:13:35.508644+0000","last_became_peered":"2023-12-16T21:13:35.508644+0000","last_unstale":"2023-12-16T21:13:37.485912+0000","last_undegraded":"2023-12-16T21:13:37.485912+0000","last_fullsized":"2023-12-16T21:13:37.485912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486627+0000","last_change":"2023-12-16T21:13:36.508270+0000","last_active":"2023-12-16T21:13:37.486627+0000","last_peered":"2023-12-16T21:13:37.486627+0000","last_clean":"2023-12-16T21:13:37.486627+0000","last_became_active":"2023-12-16T21:13:36.507998+0000","last_became_peered":"2023-12-16T21:13:36.507998+0000","last_unstale":"2023-12-16T21:13:37.486627+0000","last_undegraded":"2023-12-16T21:13:37.486627+0000","last_fullsized":"2023-12-16T21:13:37.486627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486639+0000","last_change":"2023-12-16T21:13:35.508588+0000","last_active":"2023-12-16T21:13:37.486639+0000","last_peered":"2023-12-16T21:13:37.486639+0000","last_clean":"2023-12-16T21:13:37.486639+0000","last_became_active":"2023-12-16T21:13:35.508246+0000","last_became_peered":"2023-12-16T21:13:35.508246+0000","last_unstale":"2023-12-16T21:13:37.486639+0000","last_undegraded":"2023-12-16T21:13:37.486639+0000","last_fullsized":"2023-12-16T21:13:37.486639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004928+0000","last_change":"2023-12-16T21:13:36.507570+0000","last_active":"2023-12-16T21:13:38.004928+0000","last_peered":"2023-12-16T21:13:38.004928+0000","last_clean":"2023-12-16T21:13:38.004928+0000","last_became_active":"2023-12-16T21:13:36.507226+0000","last_became_peered":"2023-12-16T21:13:36.507226+0000","last_unstale":"2023-12-16T21:13:38.004928+0000","last_undegraded":"2023-12-16T21:13:38.004928+0000","last_fullsized":"2023-12-16T21:13:38.004928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968370+0000","last_change":"2023-12-16T21:13:35.509646+0000","last_active":"2023-12-16T21:13:37.968370+0000","last_peered":"2023-12-16T21:13:37.968370+0000","last_clean":"2023-12-16T21:13:37.968370+0000","last_became_active":"2023-12-16T21:13:35.509335+0000","last_became_peered":"2023-12-16T21:13:35.509335+0000","last_unstale":"2023-12-16T21:13:37.968370+0000","last_undegraded":"2023-12-16T21:13:37.968370+0000","last_fullsized":"2023-12-16T21:13:37.968370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043143+0000","last_change":"2023-12-16T21:13:36.499896+0000","last_active":"2023-12-16T21:13:38.043143+0000","last_peered":"2023-12-16T21:13:38.043143+0000","last_clean":"2023-12-16T21:13:38.043143+0000","last_became_active":"2023-12-16T21:13:36.499625+0000","last_became_peered":"2023-12-16T21:13:36.499625+0000","last_unstale":"2023-12-16T21:13:38.043143+0000","last_undegraded":"2023-12-16T21:13:38.043143+0000","last_fullsized":"2023-12-16T21:13:38.043143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486051+0000","last_change":"2023-12-16T21:13:35.507588+0000","last_active":"2023-12-16T21:13:37.486051+0000","last_peered":"2023-12-16T21:13:37.486051+0000","last_clean":"2023-12-16T21:13:37.486051+0000","last_became_active":"2023-12-16T21:13:35.507264+0000","last_became_peered":"2023-12-16T21:13:35.507264+0000","last_unstale":"2023-12-16T21:13:37.486051+0000","last_undegraded":"2023-12-16T21:13:37.486051+0000","last_fullsized":"2023-12-16T21:13:37.486051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580181+0000","last_change":"2023-12-16T21:13:36.495697+0000","last_active":"2023-12-16T21:13:37.580181+0000","last_peered":"2023-12-16T21:13:37.580181+0000","last_clean":"2023-12-16T21:13:37.580181+0000","last_became_active":"2023-12-16T21:13:36.495099+0000","last_became_peered":"2023-12-16T21:13:36.495099+0000","last_unstale":"2023-12-16T21:13:37.580181+0000","last_undegraded":"2023-12-16T21:13:37.580181+0000","last_fullsized":"2023-12-16T21:13:37.580181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968435+0000","last_change":"2023-12-16T21:13:35.502116+0000","last_active":"2023-12-16T21:13:37.968435+0000","last_peered":"2023-12-16T21:13:37.968435+0000","last_clean":"2023-12-16T21:13:37.968435+0000","last_became_active":"2023-12-16T21:13:35.501814+0000","last_became_peered":"2023-12-16T21:13:35.501814+0000","last_unstale":"2023-12-16T21:13:37.968435+0000","last_undegraded":"2023-12-16T21:13:37.968435+0000","last_fullsized":"2023-12-16T21:13:37.968435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004596+0000","last_change":"2023-12-16T21:13:36.499212+0000","last_active":"2023-12-16T21:13:38.004596+0000","last_peered":"2023-12-16T21:13:38.004596+0000","last_clean":"2023-12-16T21:13:38.004596+0000","last_became_active":"2023-12-16T21:13:36.498745+0000","last_became_peered":"2023-12-16T21:13:36.498745+0000","last_unstale":"2023-12-16T21:13:38.004596+0000","last_undegraded":"2023-12-16T21:13:38.004596+0000","last_fullsized":"2023-12-16T21:13:38.004596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:10.671619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486589+0000","last_change":"2023-12-16T21:13:35.506529+0000","last_active":"2023-12-16T21:13:37.486589+0000","last_peered":"2023-12-16T21:13:37.486589+0000","last_clean":"2023-12-16T21:13:37.486589+0000","last_became_active":"2023-12-16T21:13:35.506231+0000","last_became_peered":"2023-12-16T21:13:35.506231+0000","last_unstale":"2023-12-16T21:13:37.486589+0000","last_undegraded":"2023-12-16T21:13:37.486589+0000","last_fullsized":"2023-12-16T21:13:37.486589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580108+0000","last_change":"2023-12-16T21:13:36.504072+0000","last_active":"2023-12-16T21:13:37.580108+0000","last_peered":"2023-12-16T21:13:37.580108+0000","last_clean":"2023-12-16T21:13:37.580108+0000","last_became_active":"2023-12-16T21:13:36.503801+0000","last_became_peered":"2023-12-16T21:13:36.503801+0000","last_unstale":"2023-12-16T21:13:37.580108+0000","last_undegraded":"2023-12-16T21:13:37.580108+0000","last_fullsized":"2023-12-16T21:13:37.580108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485664+0000","last_change":"2023-12-16T21:13:35.508204+0000","last_active":"2023-12-16T21:13:37.485664+0000","last_peered":"2023-12-16T21:13:37.485664+0000","last_clean":"2023-12-16T21:13:37.485664+0000","last_became_active":"2023-12-16T21:13:35.507929+0000","last_became_peered":"2023-12-16T21:13:35.507929+0000","last_unstale":"2023-12-16T21:13:37.485664+0000","last_undegraded":"2023-12-16T21:13:37.485664+0000","last_fullsized":"2023-12-16T21:13:37.485664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485428+0000","last_change":"2023-12-16T21:13:36.497155+0000","last_active":"2023-12-16T21:13:37.485428+0000","last_peered":"2023-12-16T21:13:37.485428+0000","last_clean":"2023-12-16T21:13:37.485428+0000","last_became_active":"2023-12-16T21:13:36.496862+0000","last_became_peered":"2023-12-16T21:13:36.496862+0000","last_unstale":"2023-12-16T21:13:37.485428+0000","last_undegraded":"2023-12-16T21:13:37.485428+0000","last_fullsized":"2023-12-16T21:13:37.485428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968533+0000","last_change":"2023-12-16T21:13:35.508118+0000","last_active":"2023-12-16T21:13:37.968533+0000","last_peered":"2023-12-16T21:13:37.968533+0000","last_clean":"2023-12-16T21:13:37.968533+0000","last_became_active":"2023-12-16T21:13:35.507827+0000","last_became_peered":"2023-12-16T21:13:35.507827+0000","last_unstale":"2023-12-16T21:13:37.968533+0000","last_undegraded":"2023-12-16T21:13:37.968533+0000","last_fullsized":"2023-12-16T21:13:37.968533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580055+0000","last_change":"2023-12-16T21:13:36.504720+0000","last_active":"2023-12-16T21:13:37.580055+0000","last_peered":"2023-12-16T21:13:37.580055+0000","last_clean":"2023-12-16T21:13:37.580055+0000","last_became_active":"2023-12-16T21:13:36.504444+0000","last_became_peered":"2023-12-16T21:13:36.504444+0000","last_unstale":"2023-12-16T21:13:37.580055+0000","last_undegraded":"2023-12-16T21:13:37.580055+0000","last_fullsized":"2023-12-16T21:13:37.580055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968696+0000","last_change":"2023-12-16T21:13:35.509590+0000","last_active":"2023-12-16T21:13:37.968696+0000","last_peered":"2023-12-16T21:13:37.968696+0000","last_clean":"2023-12-16T21:13:37.968696+0000","last_became_active":"2023-12-16T21:13:35.509278+0000","last_became_peered":"2023-12-16T21:13:35.509278+0000","last_unstale":"2023-12-16T21:13:37.968696+0000","last_undegraded":"2023-12-16T21:13:37.968696+0000","last_fullsized":"2023-12-16T21:13:37.968696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:46.659974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580433+0000","last_change":"2023-12-16T21:13:36.498151+0000","last_active":"2023-12-16T21:13:37.580433+0000","last_peered":"2023-12-16T21:13:37.580433+0000","last_clean":"2023-12-16T21:13:37.580433+0000","last_became_active":"2023-12-16T21:13:36.497907+0000","last_became_peered":"2023-12-16T21:13:36.497907+0000","last_unstale":"2023-12-16T21:13:37.580433+0000","last_undegraded":"2023-12-16T21:13:37.580433+0000","last_fullsized":"2023-12-16T21:13:37.580433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968604+0000","last_change":"2023-12-16T21:13:35.507707+0000","last_active":"2023-12-16T21:13:37.968604+0000","last_peered":"2023-12-16T21:13:37.968604+0000","last_clean":"2023-12-16T21:13:37.968604+0000","last_became_active":"2023-12-16T21:13:35.507148+0000","last_became_peered":"2023-12-16T21:13:35.507148+0000","last_unstale":"2023-12-16T21:13:37.968604+0000","last_undegraded":"2023-12-16T21:13:37.968604+0000","last_fullsized":"2023-12-16T21:13:37.968604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580539+0000","last_change":"2023-12-16T21:13:36.503790+0000","last_active":"2023-12-16T21:13:37.580539+0000","last_peered":"2023-12-16T21:13:37.580539+0000","last_clean":"2023-12-16T21:13:37.580539+0000","last_became_active":"2023-12-16T21:13:36.503514+0000","last_became_peered":"2023-12-16T21:13:36.503514+0000","last_unstale":"2023-12-16T21:13:37.580539+0000","last_undegraded":"2023-12-16T21:13:37.580539+0000","last_fullsized":"2023-12-16T21:13:37.580539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:43:28.130579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968380+0000","last_change":"2023-12-16T21:13:35.507575+0000","last_active":"2023-12-16T21:13:37.968380+0000","last_peered":"2023-12-16T21:13:37.968380+0000","last_clean":"2023-12-16T21:13:37.968380+0000","last_became_active":"2023-12-16T21:13:35.507291+0000","last_became_peered":"2023-12-16T21:13:35.507291+0000","last_unstale":"2023-12-16T21:13:37.968380+0000","last_undegraded":"2023-12-16T21:13:37.968380+0000","last_fullsized":"2023-12-16T21:13:37.968380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487138+0000","last_change":"2023-12-16T21:13:36.507654+0000","last_active":"2023-12-16T21:13:37.487138+0000","last_peered":"2023-12-16T21:13:37.487138+0000","last_clean":"2023-12-16T21:13:37.487138+0000","last_became_active":"2023-12-16T21:13:36.507245+0000","last_became_peered":"2023-12-16T21:13:36.507245+0000","last_unstale":"2023-12-16T21:13:37.487138+0000","last_undegraded":"2023-12-16T21:13:37.487138+0000","last_fullsized":"2023-12-16T21:13:37.487138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968585+0000","last_change":"2023-12-16T21:13:35.509065+0000","last_active":"2023-12-16T21:13:37.968585+0000","last_peered":"2023-12-16T21:13:37.968585+0000","last_clean":"2023-12-16T21:13:37.968585+0000","last_became_active":"2023-12-16T21:13:35.508808+0000","last_became_peered":"2023-12-16T21:13:35.508808+0000","last_unstale":"2023-12-16T21:13:37.968585+0000","last_undegraded":"2023-12-16T21:13:37.968585+0000","last_fullsized":"2023-12-16T21:13:37.968585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580284+0000","last_change":"2023-12-16T21:13:36.507427+0000","last_active":"2023-12-16T21:13:37.580284+0000","last_peered":"2023-12-16T21:13:37.580284+0000","last_clean":"2023-12-16T21:13:37.580284+0000","last_became_active":"2023-12-16T21:13:36.507170+0000","last_became_peered":"2023-12-16T21:13:36.507170+0000","last_unstale":"2023-12-16T21:13:37.580284+0000","last_undegraded":"2023-12-16T21:13:37.580284+0000","last_fullsized":"2023-12-16T21:13:37.580284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490630+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:36.490630+0000","last_peered":"2023-12-16T21:13:36.490630+0000","last_clean":"2023-12-16T21:13:36.490630+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:36.490630+0000","last_undegraded":"2023-12-16T21:13:36.490630+0000","last_fullsized":"2023-12-16T21:13:36.490630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486371+0000","last_change":"2023-12-16T21:13:35.505945+0000","last_active":"2023-12-16T21:13:37.486371+0000","last_peered":"2023-12-16T21:13:37.486371+0000","last_clean":"2023-12-16T21:13:37.486371+0000","last_became_active":"2023-12-16T21:13:35.505670+0000","last_became_peered":"2023-12-16T21:13:35.505670+0000","last_unstale":"2023-12-16T21:13:37.486371+0000","last_undegraded":"2023-12-16T21:13:37.486371+0000","last_fullsized":"2023-12-16T21:13:37.486371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485341+0000","last_change":"2023-12-16T21:13:36.507596+0000","last_active":"2023-12-16T21:13:37.485341+0000","last_peered":"2023-12-16T21:13:37.485341+0000","last_clean":"2023-12-16T21:13:37.485341+0000","last_became_active":"2023-12-16T21:13:36.507302+0000","last_became_peered":"2023-12-16T21:13:36.507302+0000","last_unstale":"2023-12-16T21:13:37.485341+0000","last_undegraded":"2023-12-16T21:13:37.485341+0000","last_fullsized":"2023-12-16T21:13:37.485341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579838+0000","last_change":"2023-12-16T21:13:35.512856+0000","last_active":"2023-12-16T21:13:37.579838+0000","last_peered":"2023-12-16T21:13:37.579838+0000","last_clean":"2023-12-16T21:13:37.579838+0000","last_became_active":"2023-12-16T21:13:35.512539+0000","last_became_peered":"2023-12-16T21:13:35.512539+0000","last_unstale":"2023-12-16T21:13:37.579838+0000","last_undegraded":"2023-12-16T21:13:37.579838+0000","last_fullsized":"2023-12-16T21:13:37.579838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004529+0000","last_change":"2023-12-16T21:13:36.503724+0000","last_active":"2023-12-16T21:13:38.004529+0000","last_peered":"2023-12-16T21:13:38.004529+0000","last_clean":"2023-12-16T21:13:38.004529+0000","last_became_active":"2023-12-16T21:13:36.503392+0000","last_became_peered":"2023-12-16T21:13:36.503392+0000","last_unstale":"2023-12-16T21:13:38.004529+0000","last_undegraded":"2023-12-16T21:13:38.004529+0000","last_fullsized":"2023-12-16T21:13:38.004529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:02.640523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968573+0000","last_change":"2023-12-16T21:13:35.506777+0000","last_active":"2023-12-16T21:13:37.968573+0000","last_peered":"2023-12-16T21:13:37.968573+0000","last_clean":"2023-12-16T21:13:37.968573+0000","last_became_active":"2023-12-16T21:13:35.506483+0000","last_became_peered":"2023-12-16T21:13:35.506483+0000","last_unstale":"2023-12-16T21:13:37.968573+0000","last_undegraded":"2023-12-16T21:13:37.968573+0000","last_fullsized":"2023-12-16T21:13:37.968573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:25.556208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042790+0000","last_change":"2023-12-16T21:13:36.496004+0000","last_active":"2023-12-16T21:13:38.042790+0000","last_peered":"2023-12-16T21:13:38.042790+0000","last_clean":"2023-12-16T21:13:38.042790+0000","last_became_active":"2023-12-16T21:13:36.495712+0000","last_became_peered":"2023-12-16T21:13:36.495712+0000","last_unstale":"2023-12-16T21:13:38.042790+0000","last_undegraded":"2023-12-16T21:13:38.042790+0000","last_fullsized":"2023-12-16T21:13:38.042790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486820+0000","last_change":"2023-12-16T21:13:35.507557+0000","last_active":"2023-12-16T21:13:37.486820+0000","last_peered":"2023-12-16T21:13:37.486820+0000","last_clean":"2023-12-16T21:13:37.486820+0000","last_became_active":"2023-12-16T21:13:35.507175+0000","last_became_peered":"2023-12-16T21:13:35.507175+0000","last_unstale":"2023-12-16T21:13:37.486820+0000","last_undegraded":"2023-12-16T21:13:37.486820+0000","last_fullsized":"2023-12-16T21:13:37.486820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:11.664911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043017+0000","last_change":"2023-12-16T21:13:36.502502+0000","last_active":"2023-12-16T21:13:38.043017+0000","last_peered":"2023-12-16T21:13:38.043017+0000","last_clean":"2023-12-16T21:13:38.043017+0000","last_became_active":"2023-12-16T21:13:36.501893+0000","last_became_peered":"2023-12-16T21:13:36.501893+0000","last_unstale":"2023-12-16T21:13:38.043017+0000","last_undegraded":"2023-12-16T21:13:38.043017+0000","last_fullsized":"2023-12-16T21:13:38.043017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:46.012193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968352+0000","last_change":"2023-12-16T21:13:35.513556+0000","last_active":"2023-12-16T21:13:37.968352+0000","last_peered":"2023-12-16T21:13:37.968352+0000","last_clean":"2023-12-16T21:13:37.968352+0000","last_became_active":"2023-12-16T21:13:35.513208+0000","last_became_peered":"2023-12-16T21:13:35.513208+0000","last_unstale":"2023-12-16T21:13:37.968352+0000","last_undegraded":"2023-12-16T21:13:37.968352+0000","last_fullsized":"2023-12-16T21:13:37.968352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486956+0000","last_change":"2023-12-16T21:13:36.504553+0000","last_active":"2023-12-16T21:13:37.486956+0000","last_peered":"2023-12-16T21:13:37.486956+0000","last_clean":"2023-12-16T21:13:37.486956+0000","last_became_active":"2023-12-16T21:13:36.504276+0000","last_became_peered":"2023-12-16T21:13:36.504276+0000","last_unstale":"2023-12-16T21:13:37.486956+0000","last_undegraded":"2023-12-16T21:13:37.486956+0000","last_fullsized":"2023-12-16T21:13:37.486956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580388+0000","last_change":"2023-12-16T21:13:35.511628+0000","last_active":"2023-12-16T21:13:37.580388+0000","last_peered":"2023-12-16T21:13:37.580388+0000","last_clean":"2023-12-16T21:13:37.580388+0000","last_became_active":"2023-12-16T21:13:35.510934+0000","last_became_peered":"2023-12-16T21:13:35.510934+0000","last_unstale":"2023-12-16T21:13:37.580388+0000","last_undegraded":"2023-12-16T21:13:37.580388+0000","last_fullsized":"2023-12-16T21:13:37.580388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:21.664872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043123+0000","last_change":"2023-12-16T21:13:36.495330+0000","last_active":"2023-12-16T21:13:38.043123+0000","last_peered":"2023-12-16T21:13:38.043123+0000","last_clean":"2023-12-16T21:13:38.043123+0000","last_became_active":"2023-12-16T21:13:36.495064+0000","last_became_peered":"2023-12-16T21:13:36.495064+0000","last_unstale":"2023-12-16T21:13:38.043123+0000","last_undegraded":"2023-12-16T21:13:38.043123+0000","last_fullsized":"2023-12-16T21:13:38.043123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043073+0000","last_change":"2023-12-16T21:13:35.508842+0000","last_active":"2023-12-16T21:13:38.043073+0000","last_peered":"2023-12-16T21:13:38.043073+0000","last_clean":"2023-12-16T21:13:38.043073+0000","last_became_active":"2023-12-16T21:13:35.508552+0000","last_became_peered":"2023-12-16T21:13:35.508552+0000","last_unstale":"2023-12-16T21:13:38.043073+0000","last_undegraded":"2023-12-16T21:13:38.043073+0000","last_fullsized":"2023-12-16T21:13:38.043073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.497136+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.496859+0000","last_became_peered":"2023-12-16T21:13:36.496859+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486007+0000","last_change":"2023-12-16T21:13:35.512677+0000","last_active":"2023-12-16T21:13:37.486007+0000","last_peered":"2023-12-16T21:13:37.486007+0000","last_clean":"2023-12-16T21:13:37.486007+0000","last_became_active":"2023-12-16T21:13:35.511936+0000","last_became_peered":"2023-12-16T21:13:35.511936+0000","last_unstale":"2023-12-16T21:13:37.486007+0000","last_undegraded":"2023-12-16T21:13:37.486007+0000","last_fullsized":"2023-12-16T21:13:37.486007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004860+0000","last_change":"2023-12-16T21:13:36.499822+0000","last_active":"2023-12-16T21:13:38.004860+0000","last_peered":"2023-12-16T21:13:38.004860+0000","last_clean":"2023-12-16T21:13:38.004860+0000","last_became_active":"2023-12-16T21:13:36.499517+0000","last_became_peered":"2023-12-16T21:13:36.499517+0000","last_unstale":"2023-12-16T21:13:38.004860+0000","last_undegraded":"2023-12-16T21:13:38.004860+0000","last_fullsized":"2023-12-16T21:13:38.004860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579837+0000","last_change":"2023-12-16T21:13:35.511783+0000","last_active":"2023-12-16T21:13:37.579837+0000","last_peered":"2023-12-16T21:13:37.579837+0000","last_clean":"2023-12-16T21:13:37.579837+0000","last_became_active":"2023-12-16T21:13:35.510752+0000","last_became_peered":"2023-12-16T21:13:35.510752+0000","last_unstale":"2023-12-16T21:13:37.579837+0000","last_undegraded":"2023-12-16T21:13:37.579837+0000","last_fullsized":"2023-12-16T21:13:37.579837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043007+0000","last_change":"2023-12-16T21:13:36.503029+0000","last_active":"2023-12-16T21:13:38.043007+0000","last_peered":"2023-12-16T21:13:38.043007+0000","last_clean":"2023-12-16T21:13:38.043007+0000","last_became_active":"2023-12-16T21:13:36.502641+0000","last_became_peered":"2023-12-16T21:13:36.502641+0000","last_unstale":"2023-12-16T21:13:38.043007+0000","last_undegraded":"2023-12-16T21:13:38.043007+0000","last_fullsized":"2023-12-16T21:13:38.043007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968311+0000","last_change":"2023-12-16T21:13:35.511492+0000","last_active":"2023-12-16T21:13:37.968311+0000","last_peered":"2023-12-16T21:13:37.968311+0000","last_clean":"2023-12-16T21:13:37.968311+0000","last_became_active":"2023-12-16T21:13:35.511118+0000","last_became_peered":"2023-12-16T21:13:35.511118+0000","last_unstale":"2023-12-16T21:13:37.968311+0000","last_undegraded":"2023-12-16T21:13:37.968311+0000","last_fullsized":"2023-12-16T21:13:37.968311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485112+0000","last_change":"2023-12-16T21:13:36.495196+0000","last_active":"2023-12-16T21:13:37.485112+0000","last_peered":"2023-12-16T21:13:37.485112+0000","last_clean":"2023-12-16T21:13:37.485112+0000","last_became_active":"2023-12-16T21:13:36.494933+0000","last_became_peered":"2023-12-16T21:13:36.494933+0000","last_unstale":"2023-12-16T21:13:37.485112+0000","last_undegraded":"2023-12-16T21:13:37.485112+0000","last_fullsized":"2023-12-16T21:13:37.485112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968321+0000","last_change":"2023-12-16T21:13:35.506652+0000","last_active":"2023-12-16T21:13:37.968321+0000","last_peered":"2023-12-16T21:13:37.968321+0000","last_clean":"2023-12-16T21:13:37.968321+0000","last_became_active":"2023-12-16T21:13:35.506349+0000","last_became_peered":"2023-12-16T21:13:35.506349+0000","last_unstale":"2023-12-16T21:13:37.968321+0000","last_undegraded":"2023-12-16T21:13:37.968321+0000","last_fullsized":"2023-12-16T21:13:37.968321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485338+0000","last_change":"2023-12-16T21:13:36.490929+0000","last_active":"2023-12-16T21:13:37.485338+0000","last_peered":"2023-12-16T21:13:37.485338+0000","last_clean":"2023-12-16T21:13:37.485338+0000","last_became_active":"2023-12-16T21:13:36.490557+0000","last_became_peered":"2023-12-16T21:13:36.490557+0000","last_unstale":"2023-12-16T21:13:37.485338+0000","last_undegraded":"2023-12-16T21:13:37.485338+0000","last_fullsized":"2023-12-16T21:13:37.485338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580200+0000","last_change":"2023-12-16T21:13:35.511062+0000","last_active":"2023-12-16T21:13:37.580200+0000","last_peered":"2023-12-16T21:13:37.580200+0000","last_clean":"2023-12-16T21:13:37.580200+0000","last_became_active":"2023-12-16T21:13:35.510780+0000","last_became_peered":"2023-12-16T21:13:35.510780+0000","last_unstale":"2023-12-16T21:13:37.580200+0000","last_undegraded":"2023-12-16T21:13:37.580200+0000","last_fullsized":"2023-12-16T21:13:37.580200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:25.707555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485439+0000","last_change":"2023-12-16T21:13:36.499546+0000","last_active":"2023-12-16T21:13:37.485439+0000","last_peered":"2023-12-16T21:13:37.485439+0000","last_clean":"2023-12-16T21:13:37.485439+0000","last_became_active":"2023-12-16T21:13:36.499236+0000","last_became_peered":"2023-12-16T21:13:36.499236+0000","last_unstale":"2023-12-16T21:13:37.485439+0000","last_undegraded":"2023-12-16T21:13:37.485439+0000","last_fullsized":"2023-12-16T21:13:37.485439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486745+0000","last_change":"2023-12-16T21:13:36.512546+0000","last_active":"2023-12-16T21:13:37.486745+0000","last_peered":"2023-12-16T21:13:37.486745+0000","last_clean":"2023-12-16T21:13:37.486745+0000","last_became_active":"2023-12-16T21:13:36.512196+0000","last_became_peered":"2023-12-16T21:13:36.512196+0000","last_unstale":"2023-12-16T21:13:37.486745+0000","last_undegraded":"2023-12-16T21:13:37.486745+0000","last_fullsized":"2023-12-16T21:13:37.486745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580134+0000","last_change":"2023-12-16T21:13:35.511999+0000","last_active":"2023-12-16T21:13:37.580134+0000","last_peered":"2023-12-16T21:13:37.580134+0000","last_clean":"2023-12-16T21:13:37.580134+0000","last_became_active":"2023-12-16T21:13:35.511760+0000","last_became_peered":"2023-12-16T21:13:35.511760+0000","last_unstale":"2023-12-16T21:13:37.580134+0000","last_undegraded":"2023-12-16T21:13:37.580134+0000","last_fullsized":"2023-12-16T21:13:37.580134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:49.269497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487061+0000","last_change":"2023-12-16T21:13:36.512673+0000","last_active":"2023-12-16T21:13:37.487061+0000","last_peered":"2023-12-16T21:13:37.487061+0000","last_clean":"2023-12-16T21:13:37.487061+0000","last_became_active":"2023-12-16T21:13:36.512185+0000","last_became_peered":"2023-12-16T21:13:36.512185+0000","last_unstale":"2023-12-16T21:13:37.487061+0000","last_undegraded":"2023-12-16T21:13:37.487061+0000","last_fullsized":"2023-12-16T21:13:37.487061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580255+0000","last_change":"2023-12-16T21:13:35.511982+0000","last_active":"2023-12-16T21:13:37.580255+0000","last_peered":"2023-12-16T21:13:37.580255+0000","last_clean":"2023-12-16T21:13:37.580255+0000","last_became_active":"2023-12-16T21:13:35.511359+0000","last_became_peered":"2023-12-16T21:13:35.511359+0000","last_unstale":"2023-12-16T21:13:37.580255+0000","last_undegraded":"2023-12-16T21:13:37.580255+0000","last_fullsized":"2023-12-16T21:13:37.580255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004907+0000","last_change":"2023-12-16T21:13:36.505890+0000","last_active":"2023-12-16T21:13:38.004907+0000","last_peered":"2023-12-16T21:13:38.004907+0000","last_clean":"2023-12-16T21:13:38.004907+0000","last_became_active":"2023-12-16T21:13:36.505414+0000","last_became_peered":"2023-12-16T21:13:36.505414+0000","last_unstale":"2023-12-16T21:13:38.004907+0000","last_undegraded":"2023-12-16T21:13:38.004907+0000","last_fullsized":"2023-12-16T21:13:38.004907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485644+0000","last_change":"2023-12-16T21:13:35.511468+0000","last_active":"2023-12-16T21:13:37.485644+0000","last_peered":"2023-12-16T21:13:37.485644+0000","last_clean":"2023-12-16T21:13:37.485644+0000","last_became_active":"2023-12-16T21:13:35.511092+0000","last_became_peered":"2023-12-16T21:13:35.511092+0000","last_unstale":"2023-12-16T21:13:37.485644+0000","last_undegraded":"2023-12-16T21:13:37.485644+0000","last_fullsized":"2023-12-16T21:13:37.485644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485618+0000","last_change":"2023-12-16T21:13:36.494416+0000","last_active":"2023-12-16T21:13:37.485618+0000","last_peered":"2023-12-16T21:13:37.485618+0000","last_clean":"2023-12-16T21:13:37.485618+0000","last_became_active":"2023-12-16T21:13:36.494121+0000","last_became_peered":"2023-12-16T21:13:36.494121+0000","last_unstale":"2023-12-16T21:13:37.485618+0000","last_undegraded":"2023-12-16T21:13:37.485618+0000","last_fullsized":"2023-12-16T21:13:37.485618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968502+0000","last_change":"2023-12-16T21:13:35.510139+0000","last_active":"2023-12-16T21:13:37.968502+0000","last_peered":"2023-12-16T21:13:37.968502+0000","last_clean":"2023-12-16T21:13:37.968502+0000","last_became_active":"2023-12-16T21:13:35.509804+0000","last_became_peered":"2023-12-16T21:13:35.509804+0000","last_unstale":"2023-12-16T21:13:37.968502+0000","last_undegraded":"2023-12-16T21:13:37.968502+0000","last_fullsized":"2023-12-16T21:13:37.968502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:56:59.813907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.255}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.31}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:44.286 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-16T21:13:44.287 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-16T21:13:44.287 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-16T21:13:44.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-16T21:13:44.287 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-16T21:13:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.429+0000 7fdf873d5640 1 Processor -- start 2023-12-16T21:13:44.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.429+0000 7fdf873d5640 1 -- start start 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf873d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf8015d2d0 0x7fdf800b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf873d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf800ce650 0x7fdf800d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf873d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf801614e0 con 0x7fdf80160f70 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf873d5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf801617c0 con 0x7fdf800ce650 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf873d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf80161aa0 con 0x7fdf8015d2d0 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf85972640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf800ce650 0x7fdf800d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf85171640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf8015d2d0 0x7fdf800b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf84970640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdf80160f70 0x7fdf800cdf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53184/0 (socket says 172.21.15.46:53184) 2023-12-16T21:13:44.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf84970640 1 -- 172.21.15.46:0/3375070816 learned_addr learned my addr 172.21.15.46:0/3375070816 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.430+0000 7fdf77fff640 1 -- 172.21.15.46:0/3375070816 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252701120 0 0) 0x7fdf801614e0 con 0x7fdf80160f70 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf77fff640 1 -- 172.21.15.46:0/3375070816 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c003520 con 0x7fdf80160f70 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf85972640 1 -- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf8015d2d0 msgr2=0x7fdf800b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf85972640 1 --2- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf8015d2d0 0x7fdf800b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf85972640 1 -- 172.21.15.46:0/3375070816 >> v1:172.21.15.121:6789/0 conn(0x7fdf80160f70 legacy=0x7fdf800cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf85972640 1 -- 172.21.15.46:0/3375070816 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80161d80 con 0x7fdf800ce650 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf85171640 1 --2- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf8015d2d0 0x7fdf800b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.431+0000 7fdf85972640 1 --2- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf800ce650 0x7fdf800d1af0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009d90 tx=0x7fdf7802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3507598a18b405 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf77fff640 1 -- 172.21.15.46:0/3375070816 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7802cc50 con 0x7fdf800ce650 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf77fff640 1 -- 172.21.15.46:0/3375070816 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7802cdb0 con 0x7fdf800ce650 2023-12-16T21:13:44.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf77fff640 1 -- 172.21.15.46:0/3375070816 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78034660 con 0x7fdf800ce650 2023-12-16T21:13:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 -- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf800ce650 msgr2=0x7fdf800d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 --2- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf800ce650 0x7fdf800d1af0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009d90 tx=0x7fdf7802bea0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 -- 172.21.15.46:0/3375070816 shutdown_connections 2023-12-16T21:13:44.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 --2- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf800ce650 0x7fdf800d1af0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 --2- 172.21.15.46:0/3375070816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf8015d2d0 0x7fdf800b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 -- 172.21.15.46:0/3375070816 >> 172.21.15.46:0/3375070816 conn(0x7fdf80072230 msgr2=0x7fdf800cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 -- 172.21.15.46:0/3375070816 shutdown_connections 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.432+0000 7fdf873d5640 1 -- 172.21.15.46:0/3375070816 wait complete. 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.433+0000 7fdf873d5640 1 Processor -- start 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.433+0000 7fdf873d5640 1 -- start start 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf873d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf800ce650 0x7fdf8015da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf873d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 0x7fdf800b5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf873d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf80160f70 0x7fdf8015f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf873d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800c7040 con 0x7fdf8015d2d0 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf873d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800c7290 con 0x7fdf80160f70 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf873d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf800c7570 con 0x7fdf800ce650 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf84970640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 0x7fdf800b5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf84970640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 0x7fdf800b5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36066/0 (socket says 172.21.15.46:36066) 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf84970640 1 -- 172.21.15.46:0/4254991979 learned_addr learned my addr 172.21.15.46:0/4254991979 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf85171640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf800ce650 0x7fdf8015da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf85972640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf80160f70 0x7fdf8015f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.434+0000 7fdf84970640 1 -- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf800ce650 msgr2=0x7fdf8015da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf84970640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf800ce650 0x7fdf8015da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf84970640 1 -- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf80160f70 msgr2=0x7fdf8015f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf84970640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf80160f70 0x7fdf8015f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf84970640 1 -- 172.21.15.46:0/4254991979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c009400 con 0x7fdf8015d2d0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf84970640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 0x7fdf800b5020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c410 tx=0x7fdf7000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70005ce0 con 0x7fdf8015d2d0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70005e40 con 0x7fdf8015d2d0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf78009a70 con 0x7fdf8015d2d0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700072f0 con 0x7fdf8015d2d0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.435+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf800c7b30 con 0x7fdf8015d2d0 2023-12-16T21:13:44.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.436+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf80069400 con 0x7fdf8015d2d0 2023-12-16T21:13:44.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.436+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdf70007510 con 0x7fdf8015d2d0 2023-12-16T21:13:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.437+0000 7fdf75ffb640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdf6406ae40 0x7fdf6406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.437+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf70088d30 con 0x7fdf8015d2d0 2023-12-16T21:13:44.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.437+0000 7fdf85171640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdf6406ae40 0x7fdf6406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.438+0000 7fdf85171640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdf6406ae40 0x7fdf6406d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009cd0 tx=0x7fdf7c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.441+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdf70053e20 con 0x7fdf8015d2d0 2023-12-16T21:13:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.671+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdf800d0aa0 con 0x7fdf8015d2d0 2023-12-16T21:13:44.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.674+0000 7fdf75ffb640 1 -- 172.21.15.46:0/4254991979 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171034 (secure 0 0 0) 0x7fdf70017020 con 0x7fdf8015d2d0 2023-12-16T21:13:44.677 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:44.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdf6406ae40 msgr2=0x7fdf6406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdf6406ae40 0x7fdf6406d300 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009cd0 tx=0x7fdf7c009190 comp rx=0 tx=0).stop 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 msgr2=0x7fdf800b5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 0x7fdf800b5020 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000c410 tx=0x7fdf7000c8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 shutdown_connections 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdf6406ae40 0x7fdf6406d300 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdf8015d2d0 0x7fdf800b5020 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf80160f70 0x7fdf8015f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 --2- 172.21.15.46:0/4254991979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf800ce650 0x7fdf8015da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 >> 172.21.15.46:0/4254991979 conn(0x7fdf80072230 msgr2=0x7fdf800cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 shutdown_connections 2023-12-16T21:13:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.677+0000 7fdf873d5640 1 -- 172.21.15.46:0/4254991979 wait complete. 2023-12-16T21:13:44.687 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":362353154},{"type":"v1","addr":"172.21.15.121:6833","nonce":362353154}]},"active_addr":"172.21.15.121:6833/362353154","active_change":"2023-12-16T21:13:15.943830+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":379053356}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":965298409}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":1162996855}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":16104984}]}]} 2023-12-16T21:13:44.691 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-16T21:13:44.691 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-16T21:13:44.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.798+0000 7f4b43894640 1 Processor -- start 2023-12-16T21:13:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.798+0000 7f4b43894640 1 -- start start 2023-12-16T21:13:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b43894640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c150940 0x7f4b3c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b43894640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b3c15dec0 0x7f4b3c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b43894640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3c0b49f0 con 0x7f4b3c150560 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b43894640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b3560 con 0x7f4b3c15dec0 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b43894640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c0b3de0 con 0x7f4b3c150940 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41e31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b3c15dec0 0x7f4b3c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41630640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4b3c150560 0x7f4b3c0b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53434/0 (socket says 172.21.15.46:53434) 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41e31640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b3c15dec0 0x7f4b3c15d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36078/0 (socket says 172.21.15.46:36078) 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41630640 1 -- 172.21.15.46:0/2448981101 learned_addr learned my addr 172.21.15.46:0/2448981101 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b40e2f640 1 --2- 172.21.15.46:0/2448981101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c150940 0x7f4b3c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41e31640 1 -- 172.21.15.46:0/2448981101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c150940 msgr2=0x7f4b3c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41e31640 1 --2- 172.21.15.46:0/2448981101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c150940 0x7f4b3c15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41e31640 1 -- 172.21.15.46:0/2448981101 >> v1:172.21.15.46:6789/0 conn(0x7f4b3c150560 legacy=0x7f4b3c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b33fff640 1 -- 172.21.15.46:0/2448981101 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346696595 0 0) 0x7f4b3c0b49f0 con 0x7f4b3c150560 2023-12-16T21:13:44.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.799+0000 7f4b41e31640 1 -- 172.21.15.46:0/2448981101 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0b40c0 con 0x7f4b3c15dec0 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.800+0000 7f4b41e31640 1 --2- 172.21.15.46:0/2448981101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15dec0 0x7f4b3c15d740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400bfa0 tx=0x7f4b3402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75e8840c110239b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.800+0000 7f4b33fff640 1 -- 172.21.15.46:0/2448981101 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3402ea40 con 0x7f4b3c15dec0 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.800+0000 7f4b33fff640 1 -- 172.21.15.46:0/2448981101 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3402eba0 con 0x7f4b3c15dec0 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.800+0000 7f4b33fff640 1 -- 172.21.15.46:0/2448981101 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34009540 con 0x7f4b3c15dec0 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 -- 172.21.15.46:0/2448981101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15dec0 msgr2=0x7f4b3c15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 --2- 172.21.15.46:0/2448981101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15dec0 0x7f4b3c15d740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400bfa0 tx=0x7f4b3402e0a0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 -- 172.21.15.46:0/2448981101 shutdown_connections 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 --2- 172.21.15.46:0/2448981101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15dec0 0x7f4b3c15d740 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 --2- 172.21.15.46:0/2448981101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c150940 0x7f4b3c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 -- 172.21.15.46:0/2448981101 >> 172.21.15.46:0/2448981101 conn(0x7f4b3c06a4d0 msgr2=0x7f4b3c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 -- 172.21.15.46:0/2448981101 shutdown_connections 2023-12-16T21:13:44.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.801+0000 7f4b43894640 1 -- 172.21.15.46:0/2448981101 wait complete. 2023-12-16T21:13:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.802+0000 7f4b43894640 1 Processor -- start 2023-12-16T21:13:44.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.802+0000 7f4b43894640 1 -- start start 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c150560 0x7f4b3c148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c15dec0 0x7f4b3c15be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 0x7f4b3c14a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c15c7c0 con 0x7f4b3c15c340 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c149630 con 0x7f4b3c150560 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c14adc0 con 0x7f4b3c15dec0 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 0x7f4b3c14a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 0x7f4b3c14a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36092/0 (socket says 172.21.15.46:36092) 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 -- 172.21.15.46:0/606425322 learned_addr learned my addr 172.21.15.46:0/606425322 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b40e2f640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c15dec0 0x7f4b3c15be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41630640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c150560 0x7f4b3c148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 -- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c15dec0 msgr2=0x7f4b3c15be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c15dec0 0x7f4b3c15be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 -- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c150560 msgr2=0x7f4b3c148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c150560 0x7f4b3c148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 -- 172.21.15.46:0/606425322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3400bc80 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b41e31640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 0x7f4b3c14a1d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400bfa0 tx=0x7f4b34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34009e00 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b34040660 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c14c990 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.803+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34040a10 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.804+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c14cf50 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.805+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c14d140 con 0x7f4b3c15c340 2023-12-16T21:13:44.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.805+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4b34040b70 con 0x7f4b3c15c340 2023-12-16T21:13:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.806+0000 7f4b31ffb640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2006ae70 0x7f4b2006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.806+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b340ae590 con 0x7f4b3c15c340 2023-12-16T21:13:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.806+0000 7f4b41630640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2006ae70 0x7f4b2006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:44.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.806+0000 7f4b41630640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2006ae70 0x7f4b2006d330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b38000f60 tx=0x7f4b38007b50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:44.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.810+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4b34079780 con 0x7f4b3c15c340 2023-12-16T21:13:44.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.991+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b3c069400 con 0x7f4b3c15c340 2023-12-16T21:13:44.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.992+0000 7f4b31ffb640 1 -- 172.21.15.46:0/606425322 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f4b3407cda0 con 0x7f4b3c15c340 2023-12-16T21:13:44.992 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:44.992 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.993+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2006ae70 msgr2=0x7f4b2006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.993+0000 7f4b43894640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2006ae70 0x7f4b2006d330 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b38000f60 tx=0x7f4b38007b50 comp rx=0 tx=0).stop 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.993+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 msgr2=0x7f4b3c14a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.993+0000 7f4b43894640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 0x7f4b3c14a1d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b3400bfa0 tx=0x7f4b34004a40 comp rx=0 tx=0).stop 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 shutdown_connections 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4b2006ae70 0x7f4b2006d330 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b3c15c340 0x7f4b3c14a1d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b3c15dec0 0x7f4b3c15be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 --2- 172.21.15.46:0/606425322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b3c150560 0x7f4b3c148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 >> 172.21.15.46:0/606425322 conn(0x7f4b3c06a4d0 msgr2=0x7f4b3c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 shutdown_connections 2023-12-16T21:13:44.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:44.994+0000 7f4b43894640 1 -- 172.21.15.46:0/606425322 wait complete. 2023-12-16T21:13:45.004 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-16T21:13:45.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:45.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.110+0000 7f044bf61640 1 Processor -- start 2023-12-16T21:13:45.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.110+0000 7f044bf61640 1 -- start start 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.111+0000 7f044bf61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440ceb70 0x7f04440cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.111+0000 7f044bf61640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04440b4370 0x7f04440b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.111+0000 7f044bf61640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0444071770 con 0x7f0444144980 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.111+0000 7f044bf61640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044415ced0 con 0x7f04440b4370 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.111+0000 7f044bf61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04440b4d20 con 0x7f04440ceb70 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f044a4fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04440b4370 0x7f04440b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f04494fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440ceb70 0x7f04440cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f0449cfd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0444144980 0x7f044415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53186/0 (socket says 172.21.15.46:53186) 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f0449cfd640 1 -- 172.21.15.46:0/3976431672 learned_addr learned my addr 172.21.15.46:0/3976431672 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f0448cfb640 1 -- 172.21.15.46:0/3976431672 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865260565 0 0) 0x7f0444071770 con 0x7f0444144980 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f0448cfb640 1 -- 172.21.15.46:0/3976431672 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0430003520 con 0x7f0444144980 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f044a4fe640 1 -- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440ceb70 msgr2=0x7f04440cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f044a4fe640 1 --2- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440ceb70 0x7f04440cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f044a4fe640 1 -- 172.21.15.46:0/3976431672 >> v1:172.21.15.121:6789/0 conn(0x7f0444144980 legacy=0x7f044415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f044a4fe640 1 -- 172.21.15.46:0/3976431672 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04440ccdc0 con 0x7f04440b4370 2023-12-16T21:13:45.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.112+0000 7f04494fc640 1 --2- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440ceb70 0x7f04440cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:45.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044a4fe640 1 --2- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440b4370 0x7f04440b47e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f043c009dc0 tx=0x7f043c02be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b9d1945ff59d6dd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f0448cfb640 1 -- 172.21.15.46:0/3976431672 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c02cc40 con 0x7f04440b4370 2023-12-16T21:13:45.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f0448cfb640 1 -- 172.21.15.46:0/3976431672 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c02cda0 con 0x7f04440b4370 2023-12-16T21:13:45.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f0448cfb640 1 -- 172.21.15.46:0/3976431672 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c034640 con 0x7f04440b4370 2023-12-16T21:13:45.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 -- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440b4370 msgr2=0x7f04440b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 --2- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440b4370 0x7f04440b47e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f043c009dc0 tx=0x7f043c02be90 comp rx=0 tx=0).stop 2023-12-16T21:13:45.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 -- 172.21.15.46:0/3976431672 shutdown_connections 2023-12-16T21:13:45.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 --2- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440b4370 0x7f04440b47e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 --2- 172.21.15.46:0/3976431672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440ceb70 0x7f04440cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 -- 172.21.15.46:0/3976431672 >> 172.21.15.46:0/3976431672 conn(0x7f044406a4d0 msgr2=0x7f044406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 -- 172.21.15.46:0/3976431672 shutdown_connections 2023-12-16T21:13:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.113+0000 7f044bf61640 1 -- 172.21.15.46:0/3976431672 wait complete. 2023-12-16T21:13:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 Processor -- start 2023-12-16T21:13:45.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 -- start start 2023-12-16T21:13:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440b4370 0x7f0444157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440ceb70 0x7f04440b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 0x7f04440b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0444139e90 con 0x7f0444144980 2023-12-16T21:13:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044415b270 con 0x7f04440ceb70 2023-12-16T21:13:45.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044bf61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0444157990 con 0x7f04440b4370 2023-12-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.115+0000 7f044a4fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 0x7f04440b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 0x7f04440b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36106/0 (socket says 172.21.15.46:36106) 2023-12-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 -- 172.21.15.46:0/847991038 learned_addr learned my addr 172.21.15.46:0/847991038 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f0449cfd640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440b4370 0x7f0444157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f04494fc640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440ceb70 0x7f04440b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 -- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440b4370 msgr2=0x7f0444157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440b4370 0x7f0444157e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 -- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440ceb70 msgr2=0x7f04440b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440ceb70 0x7f04440b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 -- 172.21.15.46:0/847991038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434009400 con 0x7f0444144980 2023-12-16T21:13:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044a4fe640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 0x7f04440b31a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c4a0 tx=0x7f043c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c03e610 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043c03e770 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c009a70 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c03eb20 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.116+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0444157060 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.117+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0444157b40 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.118+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f043c03b050 con 0x7f0444144980 2023-12-16T21:13:45.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.118+0000 7f043a7fc640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f042806ae40 0x7f042806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.118+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f043c0abcd0 con 0x7f0444144980 2023-12-16T21:13:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.118+0000 7f0449cfd640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f042806ae40 0x7f042806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.119+0000 7f0449cfd640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f042806ae40 0x7f042806d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0440005880 tx=0x7f0440009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.122+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0444157b40 con 0x7f0444144980 2023-12-16T21:13:45.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.298+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0444157b00 con 0x7f0444144980 2023-12-16T21:13:45.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.299+0000 7f043a7fc640 1 -- 172.21.15.46:0/847991038 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f043c039090 con 0x7f0444144980 2023-12-16T21:13:45.299 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:45.299 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:45.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f042806ae40 msgr2=0x7f042806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f042806ae40 0x7f042806d300 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0440005880 tx=0x7f0440009040 comp rx=0 tx=0).stop 2023-12-16T21:13:45.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 msgr2=0x7f04440b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 0x7f04440b31a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f043c02c4a0 tx=0x7f043c004a40 comp rx=0 tx=0).stop 2023-12-16T21:13:45.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 shutdown_connections 2023-12-16T21:13:45.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f042806ae40 0x7f042806d300 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0444144980 0x7f04440b31a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04440ceb70 0x7f04440b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 --2- 172.21.15.46:0/847991038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04440b4370 0x7f0444157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 >> 172.21.15.46:0/847991038 conn(0x7f044406a4d0 msgr2=0x7f044415b630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 shutdown_connections 2023-12-16T21:13:45.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.301+0000 7f044bf61640 1 -- 172.21.15.46:0/847991038 wait complete. 2023-12-16T21:13:45.312 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-16T21:13:45.312 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-16T21:13:45.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-16T21:13:45.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-16T21:13:45.313 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-16T21:13:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-16T21:13:45.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-16T21:13:45.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-16T21:13:45.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.498+0000 7f0c89168640 1 Processor -- start 2023-12-16T21:13:45.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.498+0000 7f0c89168640 1 -- start start 2023-12-16T21:13:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.499+0000 7f3542ce0640 1 Processor -- start 2023-12-16T21:13:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.499+0000 7f3542ce0640 1 -- start start 2023-12-16T21:13:45.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.499+0000 7f9431c27640 1 Processor -- start 2023-12-16T21:13:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.499+0000 7f9431c27640 1 -- start start 2023-12-16T21:13:45.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.499+0000 7f0c89168640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c840b68e0 0x7f0c840d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c89168640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0c840cb720 0x7f0c8415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c89168640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840cd770 con 0x7f0c840b2cf0 2023-12-16T21:13:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c89168640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c840ce480 con 0x7f0c840b68e0 2023-12-16T21:13:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7fbde0d99640 1 Processor -- start 2023-12-16T21:13:45.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c89168640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c840cbe80 con 0x7f0c840cb720 2023-12-16T21:13:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7fbde0d99640 1 -- start start 2023-12-16T21:13:45.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c82d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c840b68e0 0x7f0c840d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c82d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0c840b68e0 0x7f0c840d07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36116/0 (socket says 172.21.15.46:36116) 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c82d9d640 1 -- 172.21.15.46:0/4214425226 learned_addr learned my addr 172.21.15.46:0/4214425226 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f0c8259c640 1 --2- 172.21.15.46:0/4214425226 >> v2:172.21.15.121:3300/0 conn(0x7f0c840cb720 0x7f0c8415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f0c82d9d640 1 -- 172.21.15.46:0/4214425226 >> v2:172.21.15.121:3300/0 conn(0x7f0c840cb720 msgr2=0x7f0c8415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f0c82d9d640 1 --2- 172.21.15.46:0/4214425226 >> v2:172.21.15.121:3300/0 conn(0x7f0c840cb720 0x7f0c8415c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f0c82d9d640 1 -- 172.21.15.46:0/4214425226 >> v1:172.21.15.121:6789/0 conn(0x7f0c840b2cf0 legacy=0x7f0c840b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7ff76c6b2640 1 Processor -- start 2023-12-16T21:13:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f0c82d9d640 1 -- 172.21.15.46:0/4214425226 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840cbba0 con 0x7f0c840b68e0 2023-12-16T21:13:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7ff76c6b2640 1 -- start start 2023-12-16T21:13:45.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f059a153640 1 Processor -- start 2023-12-16T21:13:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f3542ce0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f353c1446a0 0x7f353c06a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f3542ce0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c1625b0 0x7f353c06ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f3542ce0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f353c162c10 con 0x7f353c06aa10 2023-12-16T21:13:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f3542ce0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f353c06aea0 con 0x7f353c1446a0 2023-12-16T21:13:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.500+0000 7f3542ce0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c1478a0 con 0x7f353c1625b0 2023-12-16T21:13:45.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f3540a7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f353c1446a0 0x7f353c06a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f353bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f353c06aa10 0x7f353c161ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53450/0 (socket says 172.21.15.46:53450) 2023-12-16T21:13:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f353bfff640 1 -- 172.21.15.46:0/4162014800 learned_addr learned my addr 172.21.15.46:0/4162014800 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f059a153640 1 -- start start 2023-12-16T21:13:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f3540a7c640 1 -- 172.21.15.46:0/4162014800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c1625b0 msgr2=0x7f353c06ee50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-16T21:13:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f3540a7c640 1 --2- 172.21.15.46:0/4162014800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c1625b0 0x7f353c06ee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.501+0000 7f3540a7c640 1 -- 172.21.15.46:0/4162014800 >> v1:172.21.15.46:6789/0 conn(0x7f353c06aa10 legacy=0x7f353c161ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f3540a7c640 1 -- 172.21.15.46:0/4162014800 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c144a80 con 0x7f353c1446a0 2023-12-16T21:13:45.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f0c82d9d640 1 --2- 172.21.15.46:0/4214425226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b68e0 0x7f0c840d07d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cebd0 tx=0x7f0c7802b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8909214a48aed57f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbde0d99640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbddc071880 0x7fbddc15ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbde0d99640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbddc0ce0f0 0x7fbddc0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbde0d99640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc0cea50 con 0x7fbddc06e2e0 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbde0d99640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbddc15dc50 con 0x7fbddc0ce0f0 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbde0d99640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbddc1596c0 con 0x7fbddc071880 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbddaffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbddc071880 0x7fbddc15ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbddaffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbddc071880 0x7fbddc15ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58160/0 (socket says 172.21.15.46:58160) 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7fbddaffd640 1 -- 172.21.15.46:0/2002196598 learned_addr learned my addr 172.21.15.46:0/2002196598 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f059a153640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0594146ba0 0x7f05940b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f9431c27640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 0x7f94240540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7ff76c6b2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 0x7ff764161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f059a153640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594072400 0x7f0594070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f059a153640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0594146f80 con 0x7f05940b42d0 2023-12-16T21:13:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f059a153640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05940ac8e0 con 0x7f0594146ba0 2023-12-16T21:13:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f059a153640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05940b47e0 con 0x7f0594072400 2023-12-16T21:13:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f9431c27640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94240549a0 0x7f9424050090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f05937fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0594146ba0 0x7f05940b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7f0c81d9b640 1 -- 172.21.15.46:0/4214425226 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78030710 con 0x7f0c840b68e0 2023-12-16T21:13:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f0c81d9b640 1 -- 172.21.15.46:0/4214425226 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7802e050 con 0x7f0c840b68e0 2023-12-16T21:13:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9431c27640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94240505d0 con 0x7f9424052ab0 2023-12-16T21:13:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f0592ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f05940b42d0 0x7f059415d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53200/0 (socket says 172.21.15.46:53200) 2023-12-16T21:13:45.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f0c81d9b640 1 -- 172.21.15.46:0/4214425226 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78030a90 con 0x7f0c840b68e0 2023-12-16T21:13:45.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9431c27640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9424007030 con 0x7f94240549a0 2023-12-16T21:13:45.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9430c25640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 0x7f94240540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f0593fff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594072400 0x7f0594070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f0592ffd640 1 -- 172.21.15.46:0/3879736597 learned_addr learned my addr 172.21.15.46:0/3879736597 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7fbddaffd640 1 -- 172.21.15.46:0/2002196598 >> v2:172.21.15.46:3300/0 conn(0x7fbddc0ce0f0 msgr2=0x7fbddc0ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9430c25640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 0x7f94240540a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51002/0 (socket says 172.21.15.46:51002) 2023-12-16T21:13:45.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9431426640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9424052ab0 0x7f9424052f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53214/0 (socket says 172.21.15.46:53214) 2023-12-16T21:13:45.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9430c25640 1 -- 172.21.15.46:0/845492030 learned_addr learned my addr 172.21.15.46:0/845492030 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f9431c27640 1 -- 172.21.15.46:0/845492030 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9424007310 con 0x7f942404fcb0 2023-12-16T21:13:45.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7fbddaffd640 1 --2- 172.21.15.46:0/2002196598 >> v2:172.21.15.46:3300/0 conn(0x7fbddc0ce0f0 0x7fbddc0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7fbddaffd640 1 -- 172.21.15.46:0/2002196598 >> v1:172.21.15.46:6789/0 conn(0x7fbddc06e2e0 legacy=0x7fbddc159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7fbddaffd640 1 -- 172.21.15.46:0/2002196598 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc15d3b0 con 0x7fbddc071880 2023-12-16T21:13:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f942bfff640 1 --2- 172.21.15.46:0/845492030 >> v2:172.21.15.46:3300/0 conn(0x7f94240549a0 0x7f9424050090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7fa12c41c640 1 Processor -- start 2023-12-16T21:13:45.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f3540a7c640 1 --2- 172.21.15.46:0/4162014800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1446a0 0x7f353c06a4d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3534009d90 tx=0x7f353402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da0a0254261db62 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7efe5759e640 1 Processor -- start 2023-12-16T21:13:45.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f353b7fe640 1 -- 172.21.15.46:0/4162014800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353402cb60 con 0x7f353c1446a0 2023-12-16T21:13:45.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0c89168640 1 -- 172.21.15.46:0/4214425226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b68e0 msgr2=0x7f0c840d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0c89168640 1 --2- 172.21.15.46:0/4214425226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b68e0 0x7f0c840d07d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cebd0 tx=0x7f0c7802b750 comp rx=0 tx=0).stop 2023-12-16T21:13:45.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f353b7fe640 1 -- 172.21.15.46:0/4162014800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353402ccc0 con 0x7f353c1446a0 2023-12-16T21:13:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7fa12c41c640 1 -- start start 2023-12-16T21:13:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0c89168640 1 -- 172.21.15.46:0/4214425226 shutdown_connections 2023-12-16T21:13:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0c89168640 1 --2- 172.21.15.46:0/4214425226 >> v2:172.21.15.121:3300/0 conn(0x7f0c840cb720 0x7f0c8415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0c89168640 1 --2- 172.21.15.46:0/4214425226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b68e0 0x7f0c840d07d0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cebd0 tx=0x7f0c7802b750 comp rx=0 tx=0).stop 2023-12-16T21:13:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0c89168640 1 -- 172.21.15.46:0/4214425226 >> 172.21.15.46:0/4214425226 conn(0x7f0c84066690 msgr2=0x7f0c8406a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f353b7fe640 1 -- 172.21.15.46:0/4162014800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534034660 con 0x7f353c1446a0 2023-12-16T21:13:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640685109 0 0) 0x7f0594146f80 con 0x7f05940b42d0 2023-12-16T21:13:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f057c003520 con 0x7f05940b42d0 2023-12-16T21:13:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3498919696 0 0) 0x7f057c003520 con 0x7f05940b42d0 2023-12-16T21:13:45.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594072400 msgr2=0x7f0594070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594072400 0x7f0594070950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 >> v2:172.21.15.121:3300/0 conn(0x7f0594146ba0 msgr2=0x7f05940b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 --2- 172.21.15.46:0/3879736597 >> v2:172.21.15.121:3300/0 conn(0x7f0594146ba0 0x7f05940b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940b4ac0 con 0x7f05940b42d0 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f0588001690 con 0x7f05940b42d0 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 >> v1:172.21.15.121:6789/0 conn(0x7f05940b42d0 legacy=0x7f059415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f057c003970 0x7f057c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f057c004310 0x7f057c0047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004e50 0x7f057c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05940b47e0 con 0x7f057c004310 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c005670 con 0x7f057c004e50 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057c00bf40 con 0x7f057c003970 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0592ffd640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f057c004310 0x7f057c0047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f05937fe640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f057c003970 0x7f057c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0593fff640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594072400 0x7f0594070950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7ff76c6b2640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff7640acdb0 0x7ff7640ad240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7ff76c6b2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764162950 con 0x7ff7640721f0 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7ff76c6b2640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff764162390 con 0x7ff7640acdb0 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.502+0000 7ff76c6b2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff764162670 con 0x7ff7640b3390 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7ff76a44e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 0x7ff764161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7ff76a44e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 0x7ff764161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51024/0 (socket says 172.21.15.46:51024) 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7ff76a44e640 1 -- 172.21.15.46:0/3829277412 learned_addr learned my addr 172.21.15.46:0/3829277412 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7ff76a44e640 1 -- 172.21.15.46:0/3829277412 >> v2:172.21.15.121:3300/0 conn(0x7ff7640acdb0 msgr2=0x7ff7640ad240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7ff76a44e640 1 --2- 172.21.15.46:0/3829277412 >> v2:172.21.15.121:3300/0 conn(0x7ff7640acdb0 0x7ff7640ad240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7ff76a44e640 1 -- 172.21.15.46:0/3829277412 >> v1:172.21.15.46:6789/0 conn(0x7ff7640721f0 legacy=0x7ff7640d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7ff76a44e640 1 -- 172.21.15.46:0/3829277412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7640cfad0 con 0x7ff7640b3390 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f942bfff640 1 -- 172.21.15.46:0/845492030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 msgr2=0x7f94240540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f942bfff640 1 --2- 172.21.15.46:0/845492030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 0x7f94240540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f942bfff640 1 -- 172.21.15.46:0/845492030 >> v1:172.21.15.121:6789/0 conn(0x7f9424052ab0 legacy=0x7f9424052f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7f942bfff640 1 -- 172.21.15.46:0/845492030 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94240075f0 con 0x7f94240549a0 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f9430c25640 1 --2- 172.21.15.46:0/845492030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 0x7f94240540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7fbddaffd640 1 --2- 172.21.15.46:0/2002196598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc071880 0x7fbddc15ccb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0710d0 tx=0x7fbdcc02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a9c9587f1c986ea server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0593fff640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004e50 0x7f057c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0592ffd640 1 -- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f057c003970 msgr2=0x7f057c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0592ffd640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f057c003970 0x7f057c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0592ffd640 1 -- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004e50 msgr2=0x7f057c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0592ffd640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004e50 0x7f057c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0592ffd640 1 -- 172.21.15.46:0/3879736597 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940b4ac0 con 0x7f057c004310 2023-12-16T21:13:45.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.503+0000 7efe5759e640 1 -- start start 2023-12-16T21:13:45.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fbdda7fc640 1 -- 172.21.15.46:0/2002196598 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc031870 con 0x7fbddc071880 2023-12-16T21:13:45.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fbdda7fc640 1 -- 172.21.15.46:0/2002196598 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0319d0 con 0x7fbddc071880 2023-12-16T21:13:45.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe5759e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efe50053bd0 0x7efe500521d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa12c41c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa124159210 0x7fa12415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7f942bfff640 1 --2- 172.21.15.46:0/845492030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94240549a0 0x7f9424050090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94180097a0 tx=0x7f941802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc86aa320834c99 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa12c41c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa1240b34a0 0x7fa1240b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe5759e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500e2670 con 0x7efe50054b20 2023-12-16T21:13:45.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe5759e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe50054fb0 con 0x7efe500e1280 2023-12-16T21:13:45.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe5759e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe50055c30 con 0x7efe50053bd0 2023-12-16T21:13:45.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa12c41c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12415a830 con 0x7fa1240b40b0 2023-12-16T21:13:45.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa12c41c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1240b4a70 con 0x7fa1240b34a0 2023-12-16T21:13:45.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa12c41c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1240cd020 con 0x7fa124159210 2023-12-16T21:13:45.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe55d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efe50054b20 0x7efe50051ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53472/0 (socket says 172.21.15.46:53472) 2023-12-16T21:13:45.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe55d9b640 1 -- 172.21.15.46:0/3878766477 learned_addr learned my addr 172.21.15.46:0/3878766477 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7efe56d9d640 1 --2- 172.21.15.46:0/3878766477 >> v2:172.21.15.121:3300/0 conn(0x7efe50053bd0 0x7efe500521d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.504+0000 7f0593fff640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004e50 0x7f057c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7ff76a44e640 1 --2- 172.21.15.46:0/3829277412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 0x7ff764161a40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009e50 tx=0x7ff75c02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=773220419b0099f1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa1299b7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa1240b40b0 0x7fa12415ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53478/0 (socket says 172.21.15.46:53478) 2023-12-16T21:13:45.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa1299b7640 1 -- 172.21.15.46:0/1940192266 learned_addr learned my addr 172.21.15.46:0/1940192266 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fbdda7fc640 1 -- 172.21.15.46:0/2002196598 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc031870 con 0x7fbddc071880 2023-12-16T21:13:45.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7f0c89168640 1 -- 172.21.15.46:0/4214425226 shutdown_connections 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7f0c89168640 1 -- 172.21.15.46:0/4214425226 wait complete. 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7ff76944c640 1 -- 172.21.15.46:0/3829277412 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c02ca10 con 0x7ff7640b3390 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 -- 172.21.15.46:0/2002196598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc071880 msgr2=0x7fbddc15ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 --2- 172.21.15.46:0/2002196598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc071880 0x7fbddc15ccb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0710d0 tx=0x7fbdcc02b910 comp rx=0 tx=0).stop 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 -- 172.21.15.46:0/2002196598 shutdown_connections 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 --2- 172.21.15.46:0/2002196598 >> v2:172.21.15.46:3300/0 conn(0x7fbddc0ce0f0 0x7fbddc0ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 --2- 172.21.15.46:0/2002196598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc071880 0x7fbddc15ccb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 -- 172.21.15.46:0/2002196598 >> 172.21.15.46:0/2002196598 conn(0x7fbddc06a4d0 msgr2=0x7fbddc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 -- 172.21.15.46:0/2002196598 shutdown_connections 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fbde0d99640 1 -- 172.21.15.46:0/2002196598 wait complete. 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 -- 172.21.15.46:0/4162014800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1446a0 msgr2=0x7f353c06a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 --2- 172.21.15.46:0/4162014800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1446a0 0x7f353c06a4d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3534009d90 tx=0x7f353402bdb0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 -- 172.21.15.46:0/4162014800 shutdown_connections 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 --2- 172.21.15.46:0/4162014800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c1625b0 0x7f353c06ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 --2- 172.21.15.46:0/4162014800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1446a0 0x7f353c06a4d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 -- 172.21.15.46:0/4162014800 >> 172.21.15.46:0/4162014800 conn(0x7f353c0b1e50 msgr2=0x7f353c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 -- 172.21.15.46:0/4162014800 shutdown_connections 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991209793 0 0) 0x7efe4c001110 con 0x7efe50054b20 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe40003520 con 0x7efe50054b20 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265060915 0 0) 0x7efe50054fb0 con 0x7efe500e1280 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c001110 con 0x7efe500e1280 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7ff76944c640 1 -- 172.21.15.46:0/3829277412 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c02cb70 con 0x7ff7640b3390 2023-12-16T21:13:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7ff76944c640 1 -- 172.21.15.46:0/3829277412 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c0343f0 con 0x7ff7640b3390 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.505+0000 7fa12a1b8640 1 --2- 172.21.15.46:0/1940192266 >> v2:172.21.15.121:3300/0 conn(0x7fa124159210 0x7fa12415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa12a9b9640 1 --2- 172.21.15.46:0/1940192266 >> v2:172.21.15.46:3300/0 conn(0x7fa1240b34a0 0x7fa1240b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa1291b6640 1 -- 172.21.15.46:0/1940192266 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472626554 0 0) 0x7fa12415a830 con 0x7fa1240b40b0 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa1291b6640 1 -- 172.21.15.46:0/1940192266 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa110003520 con 0x7fa1240b40b0 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa12a9b9640 1 -- 172.21.15.46:0/1940192266 >> v2:172.21.15.121:3300/0 conn(0x7fa124159210 msgr2=0x7fa12415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa12a9b9640 1 --2- 172.21.15.46:0/1940192266 >> v2:172.21.15.121:3300/0 conn(0x7fa124159210 0x7fa12415a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa12a9b9640 1 -- 172.21.15.46:0/1940192266 >> v1:172.21.15.46:6789/0 conn(0x7fa1240b40b0 legacy=0x7fa12415ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7fa12a9b9640 1 -- 172.21.15.46:0/1940192266 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1240cd300 con 0x7fa1240b34a0 2023-12-16T21:13:45.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7ff76c6b2640 1 -- 172.21.15.46:0/3829277412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 msgr2=0x7ff764161a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/3829277412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 0x7ff764161a40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009e50 tx=0x7ff75c02be70 comp rx=0 tx=0).stop 2023-12-16T21:13:45.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7ff76c6b2640 1 -- 172.21.15.46:0/3829277412 shutdown_connections 2023-12-16T21:13:45.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/3829277412 >> v2:172.21.15.121:3300/0 conn(0x7ff7640acdb0 0x7ff7640ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/3829277412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640b3390 0x7ff764161a40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7ff76c6b2640 1 -- 172.21.15.46:0/3829277412 >> 172.21.15.46:0/3829277412 conn(0x7ff76406b5e0 msgr2=0x7ff76406b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f942b7fe640 1 -- 172.21.15.46:0/845492030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941802d040 con 0x7f94240549a0 2023-12-16T21:13:45.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7f942b7fe640 1 -- 172.21.15.46:0/845492030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9418031810 con 0x7f94240549a0 2023-12-16T21:13:45.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7f942b7fe640 1 -- 172.21.15.46:0/845492030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418031ba0 con 0x7f94240549a0 2023-12-16T21:13:45.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7ff76c6b2640 1 -- 172.21.15.46:0/3829277412 shutdown_connections 2023-12-16T21:13:45.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7f0592ffd640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f057c004310 0x7f057c0047e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f05940ae5a0 tx=0x7f057c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a604579c3639b1a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.506+0000 7f3542ce0640 1 -- 172.21.15.46:0/4162014800 wait complete. 2023-12-16T21:13:45.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3301321404 0 0) 0x7efe4c001110 con 0x7efe500e1280 2023-12-16T21:13:45.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 >> v2:172.21.15.121:3300/0 conn(0x7efe50053bd0 msgr2=0x7efe500521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 --2- 172.21.15.46:0/3878766477 >> v2:172.21.15.121:3300/0 conn(0x7efe50053bd0 0x7efe500521d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7fa12a9b9640 1 --2- 172.21.15.46:0/1940192266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b34a0 0x7fa1240b4530 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa1240b1cd0 tx=0x7fa11c00a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519ec81ad84d0320 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 >> v1:172.21.15.46:6789/0 conn(0x7efe50054b20 legacy=0x7efe50051ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f0c89168640 1 Processor -- start 2023-12-16T21:13:45.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe50055290 con 0x7efe500e1280 2023-12-16T21:13:45.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7efe580689d0 con 0x7efe500e1280 2023-12-16T21:13:45.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058802cde0 con 0x7f057c004310 2023-12-16T21:13:45.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05880058d0 con 0x7f057c004310 2023-12-16T21:13:45.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 >> v1:172.21.15.121:6789/0 conn(0x7efe500e1280 legacy=0x7efe500534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe40004b60 0x7efe40004f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe40003760 0x7efe40003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe40004120 0x7efe4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe50055c30 con 0x7efe40004b60 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4000bec0 con 0x7efe40003760 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4000c080 con 0x7efe40004120 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5659c640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe40004b60 0x7efe40004f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe55d9b640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe40003760 0x7efe40003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5659c640 1 -- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe40004120 msgr2=0x7efe4000b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:45.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5659c640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe40004120 0x7efe4000b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5659c640 1 -- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe40003760 msgr2=0x7efe40003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7efe5659c640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe40003760 0x7efe40003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7fbde0d99640 1 Processor -- start 2023-12-16T21:13:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7efe5659c640 1 -- 172.21.15.46:0/3878766477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe50055290 con 0x7efe40004b60 2023-12-16T21:13:45.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7efe56d9d640 1 -- 172.21.15.46:0/3878766477 reap_dead start 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f05927fc640 1 -- 172.21.15.46:0/3879736597 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588005ce0 con 0x7f057c004310 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7f9431c27640 1 -- 172.21.15.46:0/845492030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94240549a0 msgr2=0x7f9424050090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7f9431c27640 1 --2- 172.21.15.46:0/845492030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94240549a0 0x7f9424050090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94180097a0 tx=0x7f941802b8b0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f9431c27640 1 -- 172.21.15.46:0/845492030 shutdown_connections 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f9431c27640 1 --2- 172.21.15.46:0/845492030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94240549a0 0x7f9424050090 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f94180097a0 tx=0x7f941802b8b0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f9431c27640 1 --2- 172.21.15.46:0/845492030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f942404fcb0 0x7f94240540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f9431c27640 1 -- 172.21.15.46:0/845492030 >> 172.21.15.46:0/845492030 conn(0x7f9424054ee0 msgr2=0x7f94240552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f9431c27640 1 -- 172.21.15.46:0/845492030 shutdown_connections 2023-12-16T21:13:45.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f9431c27640 1 -- 172.21.15.46:0/845492030 wait complete. 2023-12-16T21:13:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7fa1291b6640 1 -- 172.21.15.46:0/1940192266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c02cbe0 con 0x7fa1240b34a0 2023-12-16T21:13:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7fa1291b6640 1 -- 172.21.15.46:0/1940192266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c030070 con 0x7fa1240b34a0 2023-12-16T21:13:45.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f3542ce0640 1 Processor -- start 2023-12-16T21:13:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7fbde0d99640 1 -- start start 2023-12-16T21:13:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.508+0000 7f3542ce0640 1 -- start start 2023-12-16T21:13:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa1291b6640 1 -- 172.21.15.46:0/1940192266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c036400 con 0x7fa1240b34a0 2023-12-16T21:13:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fbde0d99640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc06e2e0 0x7fbddc1488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fbde0d99640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbddc071880 0x7fbddc15a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fbde0d99640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbddc0ce0f0 0x7fbddc1582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fbde0d99640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc148e00 con 0x7fbddc071880 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fbde0d99640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc15ad00 con 0x7fbddc06e2e0 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fbde0d99640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0ab9a0 con 0x7fbddc0ce0f0 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f059a153640 1 -- 172.21.15.46:0/3879736597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f057c004310 msgr2=0x7f057c0047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f059a153640 1 --2- 172.21.15.46:0/3879736597 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f057c004310 0x7f057c0047e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f05940ae5a0 tx=0x7f057c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f0593fff640 1 -- 172.21.15.46:0/3879736597 reap_dead start 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f059a153640 1 -- 172.21.15.46:0/3879736597 shutdown_connections 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f059a153640 1 -- 172.21.15.46:0/3879736597 >> 172.21.15.46:0/3879736597 conn(0x7f05940b0d60 msgr2=0x7f05940b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f059a153640 1 -- 172.21.15.46:0/3879736597 shutdown_connections 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.507+0000 7ff76c6b2640 1 -- 172.21.15.46:0/3829277412 wait complete. 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7ff76c6b2640 1 Processor -- start 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7ff76c6b2640 1 -- start start 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa12c41c640 1 -- 172.21.15.46:0/1940192266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b34a0 msgr2=0x7fa1240b4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa12c41c640 1 --2- 172.21.15.46:0/1940192266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b34a0 0x7fa1240b4530 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa1240b1cd0 tx=0x7fa11c00a400 comp rx=0 tx=0).stop 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa12c41c640 1 -- 172.21.15.46:0/1940192266 shutdown_connections 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa12c41c640 1 --2- 172.21.15.46:0/1940192266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b34a0 0x7fa1240b4530 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa12c41c640 1 --2- 172.21.15.46:0/1940192266 >> v2:172.21.15.121:3300/0 conn(0x7fa124159210 0x7fa12415a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7fa12c41c640 1 -- 172.21.15.46:0/1940192266 >> 172.21.15.46:0/1940192266 conn(0x7fa1240b0d60 msgr2=0x7fa1240b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7fbddaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbddc071880 0x7fbddc15a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7fbddaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbddc071880 0x7fbddc15a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36180/0 (socket says 172.21.15.46:36180) 2023-12-16T21:13:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7fbddaffd640 1 -- 172.21.15.46:0/419853646 learned_addr learned my addr 172.21.15.46:0/419853646 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f0c89168640 1 -- start start 2023-12-16T21:13:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7f3542ce0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c06aa10 0x7f353c1fa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7efe5659c640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe40004b60 0x7efe40004f60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe5804fab0 tx=0x7efe400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697375bf10a27a9f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe580ae440 con 0x7efe40004b60 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe580ae660 con 0x7efe40004b60 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7efe5559a640 1 -- 172.21.15.46:0/3878766477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe580aead0 con 0x7efe40004b60 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 -- 172.21.15.46:0/3878766477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe40004b60 msgr2=0x7efe40004f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe40004b60 0x7efe40004f60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe5804fab0 tx=0x7efe400057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7fbddb7fe640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc06e2e0 0x7fbddc1488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 -- 172.21.15.46:0/3878766477 shutdown_connections 2023-12-16T21:13:45.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 --2- 172.21.15.46:0/3878766477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe40004b60 0x7efe40004f60 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efe5804fab0 tx=0x7efe400057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 -- 172.21.15.46:0/3878766477 >> 172.21.15.46:0/3878766477 conn(0x7efe50020010 msgr2=0x7efe50056d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7f0c89168640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b2cf0 0x7f0c84146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7f3542ce0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f353c1446a0 0x7f353c15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 -- 172.21.15.46:0/3878766477 shutdown_connections 2023-12-16T21:13:45.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7efe5759e640 1 -- 172.21.15.46:0/3878766477 wait complete. 2023-12-16T21:13:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7f9431c27640 1 Processor -- start 2023-12-16T21:13:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7f3542ce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1625b0 0x7f353c164100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7f3542ce0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c15c900 con 0x7f353c1625b0 2023-12-16T21:13:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.510+0000 7fbddbfff640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbddc0ce0f0 0x7fbddc1582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f9431c27640 1 -- start start 2023-12-16T21:13:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f3542ce0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353c15c0f0 con 0x7f353c1446a0 2023-12-16T21:13:45.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f3542ce0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353c15c340 con 0x7f353c06aa10 2023-12-16T21:13:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.509+0000 7f059a153640 1 -- 172.21.15.46:0/3879736597 wait complete. 2023-12-16T21:13:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddaffd640 1 -- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbddc0ce0f0 msgr2=0x7fbddc1582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddaffd640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbddc0ce0f0 0x7fbddc1582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddaffd640 1 -- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc06e2e0 msgr2=0x7fbddc1488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddaffd640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc06e2e0 0x7fbddc1488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddaffd640 1 -- 172.21.15.46:0/419853646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc0094d0 con 0x7fbddc071880 2023-12-16T21:13:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f353bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f353c1446a0 0x7f353c15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f353bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f353c1446a0 0x7f353c15e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58238/0 (socket says 172.21.15.46:58238) 2023-12-16T21:13:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f353bfff640 1 -- 172.21.15.46:0/719817915 learned_addr learned my addr 172.21.15.46:0/719817915 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff76c6b2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7640721f0 0x7ff7641572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff76c6b2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640acdb0 0x7ff76415d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f354127d640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1625b0 0x7f353c164100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff76c6b2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7640b3390 0x7ff76415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff76c6b2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76415a910 con 0x7ff7640b3390 2023-12-16T21:13:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddb7fe640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbddc06e2e0 0x7fbddc1488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fbddaffd640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbddc071880 0x7fbddc15a7c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc000c00 tx=0x7fbdcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff76c6b2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76415abf0 con 0x7ff7640721f0 2023-12-16T21:13:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff76c6b2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76415aed0 con 0x7ff7640acdb0 2023-12-16T21:13:45.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f059a153640 1 Processor -- start 2023-12-16T21:13:45.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7fa12c41c640 1 -- 172.21.15.46:0/1940192266 shutdown_connections 2023-12-16T21:13:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f0c82d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b2cf0 0x7f0c84146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f0c82d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b2cf0 0x7f0c84146c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36194/0 (socket says 172.21.15.46:36194) 2023-12-16T21:13:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f0c82d9d640 1 -- 172.21.15.46:0/3107295423 learned_addr learned my addr 172.21.15.46:0/3107295423 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f0c89168640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c840cb720 0x7f0c840d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7ff769c4d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640acdb0 0x7ff76415d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f353bfff640 1 -- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c06aa10 msgr2=0x7f353c1fa680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:13:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f353bfff640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f353c06aa10 0x7f353c1fa680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f0c89168640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c840d1100 0x7f0c840b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f353bfff640 1 -- 172.21.15.46:0/719817915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1625b0 msgr2=0x7f353c164100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f353bfff640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f353c1625b0 0x7f353c164100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f353bfff640 1 -- 172.21.15.46:0/719817915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3534009a70 con 0x7f353c1446a0 2023-12-16T21:13:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7640721f0 0x7ff7641572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.557 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f0c8259c640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c840cb720 0x7f0c840d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7640721f0 0x7ff7641572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58246/0 (socket says 172.21.15.46:58246) 2023-12-16T21:13:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 -- 172.21.15.46:0/2913629483 learned_addr learned my addr 172.21.15.46:0/2913629483 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f0c8359e640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c840d1100 0x7f0c840b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc031870 con 0x7fbddc071880 2023-12-16T21:13:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7fbde0d99640 1 -- 172.21.15.46:0/419853646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc15af30 con 0x7fbddc071880 2023-12-16T21:13:45.558 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7fbde0d99640 1 -- 172.21.15.46:0/419853646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc15b4f0 con 0x7fbddc071880 2023-12-16T21:13:45.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840d1580 con 0x7f0c840b2cf0 2023-12-16T21:13:45.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.511+0000 7f059a153640 1 -- start start 2023-12-16T21:13:45.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76ac4f640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7640b3390 0x7ff76415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f353bfff640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f353c1446a0 0x7f353c15e8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f352c00b430 tx=0x7f352c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f9431c27640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f942404fcb0 0x7f94240e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840b6020 con 0x7f0c840cb720 2023-12-16T21:13:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 -- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640acdb0 msgr2=0x7ff76415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640acdb0 0x7ff76415d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7fa12c41c640 1 -- 172.21.15.46:0/1940192266 wait complete. 2023-12-16T21:13:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f9431c27640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424052ab0 0x7f94240d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 -- 172.21.15.46:0/2913629483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7640b3390 msgr2=0x7ff76415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff769c4d640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7640acdb0 0x7ff76415d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:45.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7640b3390 0x7ff76415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c011830 con 0x7f353c1446a0 2023-12-16T21:13:45.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f9431c27640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94240e0ad0 0x7f942404c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7ff76a44e640 1 -- 172.21.15.46:0/2913629483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c009b30 con 0x7ff7640721f0 2023-12-16T21:13:45.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.512+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f353c15c620 con 0x7f353c1446a0 2023-12-16T21:13:45.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f9431c27640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94240e06e0 con 0x7f9424052ab0 2023-12-16T21:13:45.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f9431c27640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94240d3a30 con 0x7f942404fcb0 2023-12-16T21:13:45.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f942bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424052ab0 0x7f94240d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f942bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424052ab0 0x7f94240d34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36212/0 (socket says 172.21.15.46:36212) 2023-12-16T21:13:45.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f942bfff640 1 -- 172.21.15.46:0/4038350472 learned_addr learned my addr 172.21.15.46:0/4038350472 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f353c15ede0 con 0x7f353c1446a0 2023-12-16T21:13:45.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff76ac4f640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff7640b3390 0x7ff76415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f9431c27640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942417c600 con 0x7f94240e0ad0 2023-12-16T21:13:45.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f059a153640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f059415c100 0x7f05941f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0305d0 con 0x7fbddc071880 2023-12-16T21:13:45.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc030940 con 0x7fbddc071880 2023-12-16T21:13:45.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f059a153640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f059415c4e0 0x7f059413d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f059a153640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05940cdc50 0x7f059413dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f059a153640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059415cd00 con 0x7f059415c100 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f0592ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f059415c4e0 0x7f059413d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f0592ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f059415c4e0 0x7f059413d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51074/0 (socket says 172.21.15.46:51074) 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f0592ffd640 1 -- 172.21.15.46:0/2831160040 learned_addr learned my addr 172.21.15.46:0/2831160040 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f0593fff640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05940cdc50 0x7f059413dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059413c500 con 0x7f05940cdc50 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059413c220 con 0x7f059415c4e0 2023-12-16T21:13:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84147600 con 0x7f0c840d1100 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7efe5759e640 1 Processor -- start 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7efe5759e640 1 -- start start 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff76a44e640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7640721f0 0x7ff7641572a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009e50 tx=0x7ff75c004a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c040670 con 0x7ff7640721f0 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c0407d0 con 0x7ff7640721f0 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c040b40 con 0x7ff7640721f0 2023-12-16T21:13:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7640d0b60 con 0x7ff7640721f0 2023-12-16T21:13:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff764158950 con 0x7ff7640721f0 2023-12-16T21:13:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f9431426640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94240e0ad0 0x7f942404c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f9430c25640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f942404fcb0 0x7f94240e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f942bfff640 1 -- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94240e0ad0 msgr2=0x7f942404c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f942bfff640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94240e0ad0 0x7f942404c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f942bfff640 1 -- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f942404fcb0 msgr2=0x7f94240e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f942bfff640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f942404fcb0 0x7f94240e01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f942bfff640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9418009400 con 0x7f9424052ab0 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f05937fe640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f059415c100 0x7f05941f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.513+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352c011990 con 0x7f353c1446a0 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c011830 con 0x7f353c1446a0 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7fa12c41c640 1 Processor -- start 2023-12-16T21:13:45.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7fa12c41c640 1 -- start start 2023-12-16T21:13:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7fbde0d99640 1 -- 172.21.15.46:0/419853646 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbddc070730 con 0x7fbddc071880 2023-12-16T21:13:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f351c000fc0 con 0x7f353c1446a0 2023-12-16T21:13:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe5759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe50052c40 0x7efe500ce430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7f942bfff640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424052ab0 0x7f94240d34f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f941802be40 tx=0x7f94180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f0593fff640 1 -- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f059415c4e0 msgr2=0x7f059413d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe5759e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe50053020 0x7efe500ce970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe5759e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe500223e0 0x7efe5003bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f0593fff640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f059415c4e0 0x7f059413d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f0593fff640 1 -- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f059415c100 msgr2=0x7f05941f75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe5759e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe500ceeb0 con 0x7efe50052c40 2023-12-16T21:13:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe5759e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5004de10 con 0x7efe50053020 2023-12-16T21:13:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe5759e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5004e060 con 0x7efe500223e0 2023-12-16T21:13:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f0593fff640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f059415c100 0x7f05941f75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.514+0000 7f0593fff640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059413a2d0 con 0x7f05940cdc50 2023-12-16T21:13:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418006e10 con 0x7f9424052ab0 2023-12-16T21:13:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe56d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe500223e0 0x7efe5003bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe56d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe500223e0 0x7efe5003bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51080/0 (socket says 172.21.15.46:51080) 2023-12-16T21:13:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe56d9d640 1 -- 172.21.15.46:0/268468433 learned_addr learned my addr 172.21.15.46:0/268468433 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12c41c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1240b34a0 0x7fa124155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941803ad60 con 0x7f9424052ab0 2023-12-16T21:13:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7f9431c27640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942417c8c0 con 0x7f9424052ab0 2023-12-16T21:13:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12c41c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b40b0 0x7fa12414ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12c41c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa124159210 0x7fa12414cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12c41c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa124155b60 con 0x7fa1240b40b0 2023-12-16T21:13:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12c41c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12414d850 con 0x7fa1240b34a0 2023-12-16T21:13:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12c41c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12414db30 con 0x7fa124159210 2023-12-16T21:13:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12a1b8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1240b34a0 0x7fa124155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7efe55d9b640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe50053020 0x7efe500ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12a1b8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1240b34a0 0x7fa124155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58294/0 (socket says 172.21.15.46:58294) 2023-12-16T21:13:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12a1b8640 1 -- 172.21.15.46:0/2012539608 learned_addr learned my addr 172.21.15.46:0/2012539608 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fa12a9b9640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa124159210 0x7fa12414cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7f9431c27640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942417ce80 con 0x7f9424052ab0 2023-12-16T21:13:45.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.515+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbdcc030aa0 con 0x7fbddc071880 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe56d9d640 1 -- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe50053020 msgr2=0x7efe500ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe56d9d640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe50053020 0x7efe500ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe5659c640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe50052c40 0x7efe500ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe56d9d640 1 -- 172.21.15.46:0/268468433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe50052c40 msgr2=0x7efe500ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe56d9d640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe50052c40 0x7efe500ce430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe56d9d640 1 -- 172.21.15.46:0/268468433 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5004e340 con 0x7efe500223e0 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe5659c640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efe50052c40 0x7efe500ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff75c040ca0 con 0x7ff7640721f0 2023-12-16T21:13:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fbdc3fff640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbdc806ae90 0x7fbdc806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7f0c82d9d640 1 -- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c840d1100 msgr2=0x7f0c840b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7f0c82d9d640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c840d1100 0x7f0c840b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7f0c82d9d640 1 -- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c840cb720 msgr2=0x7f0c840d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7f0c82d9d640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c840cb720 0x7f0c840d0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7f0c82d9d640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c780094d0 con 0x7f0c840b2cf0 2023-12-16T21:13:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fa12a1b8640 1 -- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa124159210 msgr2=0x7fa12414cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fa12a1b8640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa124159210 0x7fa12414cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fa12a1b8640 1 -- 172.21.15.46:0/2012539608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b40b0 msgr2=0x7fa12414ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fa12a1b8640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1240b40b0 0x7fa12414ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fa12a1b8640 1 -- 172.21.15.46:0/2012539608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c00ba90 con 0x7fa1240b34a0 2023-12-16T21:13:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbdcc02d040 con 0x7fbddc071880 2023-12-16T21:13:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fa12a1b8640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1240b34a0 0x7fa124155f50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa12000b9f0 tx=0x7fa12000beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7f0c82d9d640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b2cf0 0x7f0c84146c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c7802bce0 tx=0x7f0c780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7ff75affd640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff74806af10 0x7ff74806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fbdc3fff640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fbdc80730b0 0x7fbdc80754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 --> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdc8075b80 con 0x7fbdc80730b0 2023-12-16T21:13:45.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdcc0ae510 con 0x7fbddc071880 2023-12-16T21:13:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7efe56d9d640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe500223e0 0x7efe5003bc90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe50020710 tx=0x7efe4800ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7fbddb7fe640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbdc806ae90 0x7fbdc806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.516+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff75c0ac6f0 con 0x7ff7640721f0 2023-12-16T21:13:45.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7ff769c4d640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff74806af10 0x7ff74806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12000d990 con 0x7fa1240b34a0 2023-12-16T21:13:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12000f890 con 0x7fa1240b34a0 2023-12-16T21:13:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fbddbfff640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fbdc80730b0 0x7fbdc80754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f940c000fc0 con 0x7f9424052ab0 2023-12-16T21:13:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7ff74c0015e0 0x7ff74c003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 --> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff74c006b40 con 0x7ff74c0015e0 2023-12-16T21:13:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800ee50 con 0x7efe500223e0 2023-12-16T21:13:45.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800c860 con 0x7efe500223e0 2023-12-16T21:13:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f0593fff640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05940cdc50 0x7f059413dc30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0594067f00 tx=0x7f058c010b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5004e620 con 0x7efe500223e0 2023-12-16T21:13:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7ff76ac4f640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7ff74c0015e0 0x7ff74c003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12000fc40 con 0x7fa1240b34a0 2023-12-16T21:13:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fa12c41c640 1 -- 172.21.15.46:0/2012539608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12414de10 con 0x7fa1240b34a0 2023-12-16T21:13:45.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f0c8259c640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c840cb720 0x7f0c840d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f0c8359e640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c840d1100 0x7f0c840b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5003c1d0 con 0x7efe500223e0 2023-12-16T21:13:45.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9418044630 con 0x7f9424052ab0 2023-12-16T21:13:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c004230 con 0x7f05940cdc50 2023-12-16T21:13:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058c004390 con 0x7f05940cdc50 2023-12-16T21:13:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058c00e8e0 con 0x7f05940cdc50 2023-12-16T21:13:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059413a5b0 con 0x7f05940cdc50 2023-12-16T21:13:45.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05941f7b30 con 0x7f05940cdc50 2023-12-16T21:13:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78032990 con 0x7f0c840b2cf0 2023-12-16T21:13:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c841f7640 con 0x7f0c840b2cf0 2023-12-16T21:13:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.518+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c78032af0 con 0x7f0c840b2cf0 2023-12-16T21:13:45.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.518+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c78032e60 con 0x7f0c840b2cf0 2023-12-16T21:13:45.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800ee50 con 0x7efe500223e0 2023-12-16T21:13:45.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fa12c41c640 1 -- 172.21.15.46:0/2012539608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12414e3d0 con 0x7fa1240b34a0 2023-12-16T21:13:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.517+0000 7fbddb7fe640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbdc806ae90 0x7fbdc806d350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4001dc0 tx=0x7fbdd4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.518+0000 7ff769c4d640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff74806af10 0x7ff74806d3d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff76415b690 tx=0x7ff760008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.518+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efe3c000fc0 con 0x7efe500223e0 2023-12-16T21:13:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7fbddbfff640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fbdc80730b0 0x7fbdc80754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f352c011b00 con 0x7f353c1446a0 2023-12-16T21:13:45.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7f35397fa640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f352806af00 0x7f352806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0574000fc0 con 0x7f05940cdc50 2023-12-16T21:13:45.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa120014070 con 0x7fa1240b34a0 2023-12-16T21:13:45.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f058c00ea40 con 0x7f05940cdc50 2023-12-16T21:13:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f352c058d40 con 0x7f353c1446a0 2023-12-16T21:13:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f35397fa640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f3528073c30 0x7f35280760d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 --> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35280767a0 con 0x7f3528073c30 2023-12-16T21:13:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f352c011830 con 0x7f353c1446a0 2023-12-16T21:13:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.519+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7efe4800c9d0 con 0x7efe500223e0 2023-12-16T21:13:45.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f941804d450 con 0x7f9424052ab0 2023-12-16T21:13:45.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f354127d640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f3528073c30 0x7f35280760d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe46ffd640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efe3806aea0 0x7efe3806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7fa11affd640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa10806af10 0x7fa10806d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== osd.0 v2:172.21.15.46:6800/2381230451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fbdc8075b80 con 0x7fbdc80730b0 2023-12-16T21:13:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe480891d0 con 0x7efe500223e0 2023-12-16T21:13:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa120088bf0 con 0x7fa1240b34a0 2023-12-16T21:13:45.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe5659c640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efe3806aea0 0x7efe3806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7fa1299b7640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa10806af10 0x7fa10806d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe46ffd640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7efe38073040 0x7efe38075460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 --> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe38075b10 con 0x7efe38073040 2023-12-16T21:13:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe48054200 con 0x7efe500223e0 2023-12-16T21:13:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f3540a7c640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f352806af00 0x7f352806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7efe55d9b640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7efe38073040 0x7efe38075460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f94297fa640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f941006ae90 0x7f941006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f057bfff640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f058006af10 0x7f058006d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.520+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f058c088fb0 con 0x7f05940cdc50 2023-12-16T21:13:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f057bfff640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7f05800730b0 0x7f05800754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0580075b80 con 0x7f05800730b0 2023-12-16T21:13:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f058c004500 con 0x7f05940cdc50 2023-12-16T21:13:45.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f9430c25640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f941006ae90 0x7f941006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7fa1299b7640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa10806af10 0x7fa10806d3d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1140048d0 tx=0x7fa114005be0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7efe5659c640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efe3806aea0 0x7efe3806d360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe400057a0 tx=0x7efe580aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f05937fe640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f058006af10 0x7f058006d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f0592ffd640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7f05800730b0 0x7f05800754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94180abed0 con 0x7f9424052ab0 2023-12-16T21:13:45.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7ff76ac4f640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7ff74c0015e0 0x7ff74c003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f354127d640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f3528073c30 0x7f35280760d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7efe55d9b640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7efe38073040 0x7efe38075460 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c840b56f0 con 0x7f0c840b2cf0 2023-12-16T21:13:45.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f94297fa640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f94100730b0 0x7f94100754d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9410075b80 con 0x7f94100730b0 2023-12-16T21:13:45.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.521+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94180ac240 con 0x7f9424052ab0 2023-12-16T21:13:45.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f9430c25640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f941006ae90 0x7f941006d350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f942000aa70 tx=0x7f9420008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f9431426640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f94100730b0 0x7f94100754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7fa12c41c640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7fa124060f20 0x7fa124061300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7fa12c41c640 1 -- 172.21.15.46:0/2012539608 --> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1240cd8c0 con 0x7fa124060f20 2023-12-16T21:13:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f3540a7c640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f352806af00 0x7f352806d3c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3534036000 tx=0x7f3534030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7fa12a9b9640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7fa124060f20 0x7fa124061300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f05937fe640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f058006af10 0x7f058006d3d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0584014450 tx=0x7f058400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f9431426640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f94100730b0 0x7f94100754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f0592ffd640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7f05800730b0 0x7f05800754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== osd.6 v2:172.21.15.121:6814/3722378128 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff74c006b40 con 0x7ff74c0015e0 2023-12-16T21:13:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0c840b6cc0 con 0x7f0c840b2cf0 2023-12-16T21:13:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0c78039040 con 0x7f0c840b2cf0 2023-12-16T21:13:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.522+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== osd.5 v2:172.21.15.121:6808/1440061954 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7efe38075b10 con 0x7efe38073040 2023-12-16T21:13:45.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7fa12a9b9640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7fa124060f20 0x7fa124061300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== osd.2 v2:172.21.15.46:6816/2454235421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0580075b80 con 0x7f05800730b0 2023-12-16T21:13:45.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== osd.1 v2:172.21.15.46:6808/2214591015 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9410075b80 con 0x7f94100730b0 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c6b7fe640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0c7006ae40 0x7f0c7006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c780aa360 con 0x7f0c840b2cf0 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c8259c640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0c7006ae40 0x7f0c7006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c6b7fe640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f0c70073060 0x7f0c70075480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c70075b30 con 0x7f0c70073060 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c780aa660 con 0x7f0c840b2cf0 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.524+0000 7f0c8359e640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f0c70073060 0x7f0c70075480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.524+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== osd.7 v2:172.21.15.121:6800/2871933776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa11c036400 con 0x7fa124060f20 2023-12-16T21:13:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.524+0000 7f0c8359e640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f0c70073060 0x7f0c70075480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.526+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== osd.3 v2:172.21.15.46:6824/1199415121 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0c70075b30 con 0x7f0c70073060 2023-12-16T21:13:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.523+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== osd.4 v2:172.21.15.121:6824/234804465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f35280767a0 con 0x7f3528073c30 2023-12-16T21:13:45.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.536+0000 7f0c8259c640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0c7006ae40 0x7f0c7006d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cd970 tx=0x7f0c74008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.553+0000 7fbde0d99640 1 -- 172.21.15.46:0/419853646 --> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbddc1598d0 con 0x7fbdc80730b0 2023-12-16T21:13:45.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.554+0000 7fbdc3fff640 1 -- 172.21.15.46:0/419853646 <== osd.0 v2:172.21.15.46:6800/2381230451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbdd0009560 con 0x7fbdc80730b0 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.554+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fbdc80730b0 msgr2=0x7fbdc80754d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.554+0000 7fbdc1ffb640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fbdc80730b0 0x7fbdc80754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbdc806ae90 msgr2=0x7fbdc806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbdc1ffb640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbdc806ae90 0x7fbdc806d350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbdd4001dc0 tx=0x7fbdd4009040 comp rx=0 tx=0).stop 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbddc071880 msgr2=0x7fbddc15a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbdc1ffb640 1 --2- 172.21.15.46:0/419853646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbddc071880 0x7fbddc15a7c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc000c00 tx=0x7fbdcc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbddbfff640 1 -- 172.21.15.46:0/419853646 reap_dead start 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 shutdown_connections 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.555+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 >> 172.21.15.46:0/419853646 conn(0x7fbddc06a4d0 msgr2=0x7fbddc06d170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.556+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 shutdown_connections 2023-12-16T21:13:45.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.556+0000 7fbdc1ffb640 1 -- 172.21.15.46:0/419853646 wait complete. 2023-12-16T21:13:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.556+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 --> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff74c005c20 con 0x7ff74c0015e0 2023-12-16T21:13:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.556+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 --> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f940c002cb0 con 0x7f94100730b0 2023-12-16T21:13:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.556+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 --> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f351c002c20 con 0x7f3528073c30 2023-12-16T21:13:45.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.557+0000 7f35397fa640 1 -- 172.21.15.46:0/719817915 <== osd.4 v2:172.21.15.121:6824/234804465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f351c002c20 con 0x7f3528073c30 2023-12-16T21:13:45.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.557+0000 7f94297fa640 1 -- 172.21.15.46:0/4038350472 <== osd.1 v2:172.21.15.46:6808/2214591015 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f940c002cb0 con 0x7f94100730b0 2023-12-16T21:13:45.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.557+0000 7ff75affd640 1 -- 172.21.15.46:0/2913629483 <== osd.6 v2:172.21.15.121:6814/3722378128 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff74c005c20 con 0x7ff74c0015e0 2023-12-16T21:13:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f94100730b0 msgr2=0x7f94100754d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f94100730b0 0x7f94100754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7fa12c41c640 1 -- 172.21.15.46:0/2012539608 --> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa1241595f0 con 0x7fa124060f20 2023-12-16T21:13:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f3528073c30 msgr2=0x7f35280760d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f3528073c30 0x7f35280760d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f352806af00 msgr2=0x7f352806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f352806af00 0x7f352806d3c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3534036000 tx=0x7f3534030000 comp rx=0 tx=0).stop 2023-12-16T21:13:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f353c1446a0 msgr2=0x7f353c15e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 --2- 172.21.15.46:0/719817915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f353c1446a0 0x7f353c15e8a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f352c00b430 tx=0x7f352c00b8f0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f354127d640 1 -- 172.21.15.46:0/719817915 reap_dead start 2023-12-16T21:13:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 --> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0574002cb0 con 0x7f05800730b0 2023-12-16T21:13:45.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 shutdown_connections 2023-12-16T21:13:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f941006ae90 msgr2=0x7f941006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f941006ae90 0x7f941006d350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f942000aa70 tx=0x7f9420008040 comp rx=0 tx=0).stop 2023-12-16T21:13:45.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 >> 172.21.15.46:0/719817915 conn(0x7f353c0b1e50 msgr2=0x7f353c144c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424052ab0 msgr2=0x7f94240d34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 --2- 172.21.15.46:0/4038350472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9424052ab0 0x7f94240d34f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f941802be40 tx=0x7f94180046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9431426640 1 -- 172.21.15.46:0/4038350472 reap_dead start 2023-12-16T21:13:45.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7fa11affd640 1 -- 172.21.15.46:0/2012539608 <== osd.7 v2:172.21.15.121:6800/2871933776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa11c00b040 con 0x7fa124060f20 2023-12-16T21:13:45.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 shutdown_connections 2023-12-16T21:13:45.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 >> 172.21.15.46:0/4038350472 conn(0x7f9424054ee0 msgr2=0x7f9424004e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 shutdown_connections 2023-12-16T21:13:45.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f3542ce0640 1 -- 172.21.15.46:0/719817915 wait complete. 2023-12-16T21:13:45.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f057bfff640 1 -- 172.21.15.46:0/2831160040 <== osd.2 v2:172.21.15.46:6816/2454235421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0574002cb0 con 0x7f05800730b0 2023-12-16T21:13:45.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 shutdown_connections 2023-12-16T21:13:45.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.558+0000 7f9416ffd640 1 -- 172.21.15.46:0/4038350472 wait complete. 2023-12-16T21:13:45.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7ff74c0015e0 msgr2=0x7ff74c003aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7ff74c0015e0 0x7ff74c003aa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7fa124060f20 msgr2=0x7fa124061300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa118ff9640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7fa124060f20 0x7fa124061300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7f05800730b0 msgr2=0x7f05800754d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f059a153640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7f05800730b0 0x7f05800754d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f058006af10 msgr2=0x7f058006d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f059a153640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f058006af10 0x7f058006d3d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0584014450 tx=0x7f058400b040 comp rx=0 tx=0).stop 2023-12-16T21:13:45.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05940cdc50 msgr2=0x7f059413dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa10806af10 msgr2=0x7fa10806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa118ff9640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa10806af10 0x7fa10806d3d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa1140048d0 tx=0x7fa114005be0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1240b34a0 msgr2=0x7fa124155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa118ff9640 1 --2- 172.21.15.46:0/2012539608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1240b34a0 0x7fa124155f50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa12000b9f0 tx=0x7fa12000beb0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7fa12a9b9640 1 -- 172.21.15.46:0/2012539608 reap_dead start 2023-12-16T21:13:45.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f059a153640 1 --2- 172.21.15.46:0/2831160040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05940cdc50 0x7f059413dc30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0594067f00 tx=0x7f058c010b60 comp rx=0 tx=0).stop 2023-12-16T21:13:45.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff74806af10 msgr2=0x7ff74806d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff74806af10 0x7ff74806d3d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff76415b690 tx=0x7ff760008040 comp rx=0 tx=0).stop 2023-12-16T21:13:45.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7640721f0 msgr2=0x7ff7641572a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 --2- 172.21.15.46:0/2913629483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7640721f0 0x7ff7641572a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009e50 tx=0x7ff75c004a70 comp rx=0 tx=0).stop 2023-12-16T21:13:45.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76ac4f640 1 -- 172.21.15.46:0/2913629483 reap_dead start 2023-12-16T21:13:45.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 shutdown_connections 2023-12-16T21:13:45.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 >> 172.21.15.46:0/2913629483 conn(0x7ff76406b5e0 msgr2=0x7ff7640aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7f0593fff640 1 -- 172.21.15.46:0/2831160040 reap_dead start 2023-12-16T21:13:45.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.559+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 shutdown_connections 2023-12-16T21:13:45.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7ff76c6b2640 1 -- 172.21.15.46:0/2913629483 wait complete. 2023-12-16T21:13:45.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 shutdown_connections 2023-12-16T21:13:45.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 >> 172.21.15.46:0/2831160040 conn(0x7f05940b0d60 msgr2=0x7f05940af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 shutdown_connections 2023-12-16T21:13:45.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 >> 172.21.15.46:0/2012539608 conn(0x7fa1240b0d60 msgr2=0x7fa124158150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 --> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efe3c002c20 con 0x7efe38073040 2023-12-16T21:13:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 shutdown_connections 2023-12-16T21:13:45.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7fa118ff9640 1 -- 172.21.15.46:0/2012539608 wait complete. 2023-12-16T21:13:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 shutdown_connections 2023-12-16T21:13:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7f059a153640 1 -- 172.21.15.46:0/2831160040 wait complete. 2023-12-16T21:13:45.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 --> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0c8415c7f0 con 0x7f0c70073060 2023-12-16T21:13:45.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.560+0000 7efe46ffd640 1 -- 172.21.15.46:0/268468433 <== osd.5 v2:172.21.15.121:6808/1440061954 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efe3c002c20 con 0x7efe38073040 2023-12-16T21:13:45.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7efe38073040 msgr2=0x7efe38075460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe5759e640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7efe38073040 0x7efe38075460 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efe3806aea0 msgr2=0x7efe3806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe5759e640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efe3806aea0 0x7efe3806d360 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efe400057a0 tx=0x7efe580aa000 comp rx=0 tx=0).stop 2023-12-16T21:13:45.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe500223e0 msgr2=0x7efe5003bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe5759e640 1 --2- 172.21.15.46:0/268468433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe500223e0 0x7efe5003bc90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efe50020710 tx=0x7efe4800ec00 comp rx=0 tx=0).stop 2023-12-16T21:13:45.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7efe56d9d640 1 -- 172.21.15.46:0/268468433 reap_dead start 2023-12-16T21:13:45.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.561+0000 7f0c6b7fe640 1 -- 172.21.15.46:0/3107295423 <== osd.3 v2:172.21.15.46:6824/1199415121 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0c7c0020e0 con 0x7f0c70073060 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f0c70073060 msgr2=0x7f0c70075480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f0c70073060 0x7f0c70075480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0c7006ae40 msgr2=0x7f0c7006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0c7006ae40 0x7f0c7006d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0c840cd970 tx=0x7f0c74008040 comp rx=0 tx=0).stop 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b2cf0 msgr2=0x7f0c84146c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 --2- 172.21.15.46:0/3107295423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0c840b2cf0 0x7f0c84146c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c7802bce0 tx=0x7f0c780046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c8359e640 1 -- 172.21.15.46:0/3107295423 reap_dead start 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 shutdown_connections 2023-12-16T21:13:45.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 >> 172.21.15.46:0/3107295423 conn(0x7f0c84066690 msgr2=0x7f0c8415cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 shutdown_connections 2023-12-16T21:13:45.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 >> 172.21.15.46:0/268468433 conn(0x7efe50020010 msgr2=0x7efe5000c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.563+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 shutdown_connections 2023-12-16T21:13:45.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 shutdown_connections 2023-12-16T21:13:45.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.562+0000 7efe5759e640 1 -- 172.21.15.46:0/268468433 wait complete. 2023-12-16T21:13:45.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.563+0000 7f0c89168640 1 -- 172.21.15.46:0/3107295423 wait complete. 2023-12-16T21:13:45.641 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:45.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-16T21:13:45.643 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:45.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-16T21:13:45.644 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:45.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-16T21:13:45.645 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:45.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-16T21:13:45.646 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:45.646 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-16T21:13:45.647 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:45.647 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-16T21:13:45.648 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:45.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-16T21:13:45.649 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-12-16T21:13:45.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-16T21:13:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f96fbfff640 1 Processor -- start 2023-12-16T21:13:45.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f96fbfff640 1 -- start start 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.864+0000 7f90d68e5640 1 Processor -- start 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.865+0000 7f90d68e5640 1 -- start start 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90d68e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90d0161720 0x7f90d0070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90d68e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d00b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90d68e5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d00726a0 con 0x7f90d00b3e40 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90d68e5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d0161b00 con 0x7f90d0161720 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90d68e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00b16c0 con 0x7f90d00b0cf0 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90d0161720 0x7f90d0070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90cf7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f90d00b3e40 0x7f90d00b2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53486/0 (socket says 172.21.15.46:53486) 2023-12-16T21:13:45.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.866+0000 7f90cf7fe640 1 -- 172.21.15.46:0/291784316 learned_addr learned my addr 172.21.15.46:0/291784316 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.867+0000 7f90d4e82640 1 --2- 172.21.15.46:0/291784316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d00b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f96fbfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f40e1010 0x7f96f4034060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f96fbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96f404ea90 0x7f96f4039eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f96fbfff640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f4033e10 con 0x7f96f403a3f0 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f96fbfff640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f40060e0 con 0x7f96f404ea90 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f96fbfff640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f404f4d0 con 0x7f96f40e1010 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f9701a4f640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96f403a3f0 0x7f96f4037ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53494/0 (socket says 172.21.15.46:53494) 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f9701a4f640 1 -- 172.21.15.46:0/3071590618 learned_addr learned my addr 172.21.15.46:0/3071590618 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f96faffd640 1 --2- 172.21.15.46:0/3071590618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f40e1010 0x7f96f4034060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f90d4e82640 1 -- 172.21.15.46:0/291784316 >> v2:172.21.15.46:3300/0 conn(0x7f90d0161720 msgr2=0x7f90d0070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f90d4e82640 1 --2- 172.21.15.46:0/291784316 >> v2:172.21.15.46:3300/0 conn(0x7f90d0161720 0x7f90d0070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f90d4e82640 1 -- 172.21.15.46:0/291784316 >> v1:172.21.15.46:6789/0 conn(0x7f90d00b3e40 legacy=0x7f90d00b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.868+0000 7f90d4e82640 1 -- 172.21.15.46:0/291784316 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d00b48e0 con 0x7f90d00b0cf0 2023-12-16T21:13:45.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7f96faffd640 1 -- 172.21.15.46:0/3071590618 >> v2:172.21.15.46:3300/0 conn(0x7f96f404ea90 msgr2=0x7f96f4039eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7f71f5397640 1 Processor -- start 2023-12-16T21:13:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7f96faffd640 1 --2- 172.21.15.46:0/3071590618 >> v2:172.21.15.46:3300/0 conn(0x7f96f404ea90 0x7f96f4039eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7f96faffd640 1 -- 172.21.15.46:0/3071590618 >> v1:172.21.15.46:6789/0 conn(0x7f96f403a3f0 legacy=0x7f96f4037ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7f96faffd640 1 -- 172.21.15.46:0/3071590618 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f40e5720 con 0x7f96f40e1010 2023-12-16T21:13:45.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7fdc89885640 1 Processor -- start 2023-12-16T21:13:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f96faffd640 1 --2- 172.21.15.46:0/3071590618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f40e1010 0x7f96f4034060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0097a0 tx=0x7f96ec02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb875bbb7914443f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7fdc89885640 1 -- start start 2023-12-16T21:13:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.869+0000 7f71f5397640 1 -- start start 2023-12-16T21:13:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f90d4e82640 1 --2- 172.21.15.46:0/291784316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d00b1180 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009d90 tx=0x7f90c802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86785732f36aa338 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71f5397640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f71f00b0cf0 0x7f71f00b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71f5397640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71f00b2060 0x7f71f00b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71f5397640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f0162d80 con 0x7f71f0158690 2023-12-16T21:13:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71f5397640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00b4c00 con 0x7f71f00b2060 2023-12-16T21:13:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71f5397640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00b16c0 con 0x7f71f00b0cf0 2023-12-16T21:13:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f90ceffd640 1 -- 172.21.15.46:0/291784316 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c802cc50 con 0x7f90d00b0cf0 2023-12-16T21:13:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f90ceffd640 1 -- 172.21.15.46:0/291784316 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c802cdb0 con 0x7f90d00b0cf0 2023-12-16T21:13:45.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f90ceffd640 1 -- 172.21.15.46:0/291784316 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8034660 con 0x7f90d00b0cf0 2023-12-16T21:13:45.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71ef7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71f00b2060 0x7f71f00b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71ef7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71f00b2060 0x7f71f00b24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36262/0 (socket says 172.21.15.46:36262) 2023-12-16T21:13:45.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.870+0000 7f71ef7fe640 1 -- 172.21.15.46:0/3222447739 learned_addr learned my addr 172.21.15.46:0/3222447739 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc89885640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdc7c0e3150 0x7fdc7c03e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc89885640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc7c0e7990 0x7fdc7c058450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc89885640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c057fc0 con 0x7fdc7c058990 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc89885640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0573b0 con 0x7fdc7c0e7990 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc89885640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c057690 con 0x7fdc7c0e3150 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f71ef7fe640 1 -- 172.21.15.46:0/3222447739 >> v2:172.21.15.121:3300/0 conn(0x7f71f00b0cf0 msgr2=0x7f71f00b1180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f71ef7fe640 1 --2- 172.21.15.46:0/3222447739 >> v2:172.21.15.121:3300/0 conn(0x7f71f00b0cf0 0x7f71f00b1180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f71ef7fe640 1 -- 172.21.15.46:0/3222447739 >> v1:172.21.15.46:6789/0 conn(0x7f71f0158690 legacy=0x7f71f00b44f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f71ef7fe640 1 -- 172.21.15.46:0/3222447739 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f00705a0 con 0x7f71f00b2060 2023-12-16T21:13:45.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdc7c0e3150 0x7fdc7c03e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdc7c0e3150 0x7fdc7c03e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58320/0 (socket says 172.21.15.46:58320) 2023-12-16T21:13:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 -- 172.21.15.46:0/2273694670 learned_addr learned my addr 172.21.15.46:0/2273694670 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc83fff640 1 --2- 172.21.15.46:0/2273694670 >> v2:172.21.15.46:3300/0 conn(0x7fdc7c0e7990 0x7fdc7c058450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f90d68e5640 1 -- 172.21.15.46:0/291784316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 msgr2=0x7f90d00b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f90d68e5640 1 --2- 172.21.15.46:0/291784316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d00b1180 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009d90 tx=0x7f90c802bea0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc837fe640 1 -- 172.21.15.46:0/2273694670 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3124732685 0 0) 0x7fdc7c057fc0 con 0x7fdc7c058990 2023-12-16T21:13:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc837fe640 1 -- 172.21.15.46:0/2273694670 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc74003520 con 0x7fdc7c058990 2023-12-16T21:13:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 -- 172.21.15.46:0/2273694670 >> v2:172.21.15.46:3300/0 conn(0x7fdc7c0e7990 msgr2=0x7fdc7c058450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 --2- 172.21.15.46:0/2273694670 >> v2:172.21.15.46:3300/0 conn(0x7fdc7c0e7990 0x7fdc7c058450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 -- 172.21.15.46:0/2273694670 >> v1:172.21.15.121:6789/0 conn(0x7fdc7c058990 legacy=0x7fdc7c053c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fdc88883640 1 -- 172.21.15.46:0/2273694670 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c057970 con 0x7fdc7c0e3150 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f90d68e5640 1 -- 172.21.15.46:0/291784316 shutdown_connections 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f90d68e5640 1 --2- 172.21.15.46:0/291784316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d00b1180 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f96f9ffb640 1 -- 172.21.15.46:0/3071590618 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec030710 con 0x7f96f40e1010 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f96f9ffb640 1 -- 172.21.15.46:0/3071590618 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ec02e050 con 0x7f96f40e1010 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7f96f9ffb640 1 -- 172.21.15.46:0/3071590618 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96ec030ad0 con 0x7f96f40e1010 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 -- 172.21.15.46:0/3071590618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f40e1010 msgr2=0x7f96f4034060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 --2- 172.21.15.46:0/3071590618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f40e1010 0x7f96f4034060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0097a0 tx=0x7f96ec02b710 comp rx=0 tx=0).stop 2023-12-16T21:13:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.871+0000 7fa3aad67640 1 Processor -- start 2023-12-16T21:13:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7fa3aad67640 1 -- start start 2023-12-16T21:13:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 -- 172.21.15.46:0/3071590618 shutdown_connections 2023-12-16T21:13:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 --2- 172.21.15.46:0/3071590618 >> v2:172.21.15.46:3300/0 conn(0x7f96f404ea90 0x7f96f4039eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f90d68e5640 1 --2- 172.21.15.46:0/291784316 >> v2:172.21.15.46:3300/0 conn(0x7f90d0161720 0x7f90d0070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f90d68e5640 1 -- 172.21.15.46:0/291784316 >> 172.21.15.46:0/291784316 conn(0x7f90d006a4d0 msgr2=0x7f90d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f90d68e5640 1 -- 172.21.15.46:0/291784316 shutdown_connections 2023-12-16T21:13:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f90d68e5640 1 -- 172.21.15.46:0/291784316 wait complete. 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 --2- 172.21.15.46:0/3071590618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f40e1010 0x7f96f4034060 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 -- 172.21.15.46:0/3071590618 >> 172.21.15.46:0/3071590618 conn(0x7f96f4054c70 msgr2=0x7f96f4055080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.872+0000 7f96fbfff640 1 -- 172.21.15.46:0/3071590618 shutdown_connections 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fdc88883640 1 --2- 172.21.15.46:0/2273694670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c0e3150 0x7fdc7c03e7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c010180 tx=0x7fdc7802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8e17e49d4d12a39 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fdc837fe640 1 -- 172.21.15.46:0/2273694670 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78031870 con 0x7fdc7c0e3150 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fdc837fe640 1 -- 172.21.15.46:0/2273694670 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc780319d0 con 0x7fdc7c0e3150 2023-12-16T21:13:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fdc837fe640 1 -- 172.21.15.46:0/2273694670 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc78031870 con 0x7fdc7c0e3150 2023-12-16T21:13:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7f71ef7fe640 1 --2- 172.21.15.46:0/3222447739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f00b2060 0x7f71f00b24d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f71f015c230 tx=0x7f71e402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa3fe8fd0ae1764 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7f71edffb640 1 -- 172.21.15.46:0/3222447739 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4031890 con 0x7f71f00b2060 2023-12-16T21:13:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7f71edffb640 1 -- 172.21.15.46:0/3222447739 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e40319f0 con 0x7f71f00b2060 2023-12-16T21:13:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7f71edffb640 1 -- 172.21.15.46:0/3222447739 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4031890 con 0x7f71f00b2060 2023-12-16T21:13:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fcb34494640 1 Processor -- start 2023-12-16T21:13:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7f90d68e5640 1 Processor -- start 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7f4489f25640 1 Processor -- start 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7f4489f25640 1 -- start start 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3aad67640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa39c0343d0 0x7fa39c053600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3aad67640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c053b40 0x7fa39c0e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3aad67640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39c034bb0 con 0x7fa39c0e76b0 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3aad67640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c0348d0 con 0x7fa39c0343d0 2023-12-16T21:13:45.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3aad67640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c054010 con 0x7fa39c053b40 2023-12-16T21:13:45.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3a9d65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa39c0e76b0 0x7fa39c052ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53508/0 (socket says 172.21.15.46:53508) 2023-12-16T21:13:45.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7fa3a9d65640 1 -- 172.21.15.46:0/1289483691 learned_addr learned my addr 172.21.15.46:0/1289483691 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:45.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fa3aa566640 1 --2- 172.21.15.46:0/1289483691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c053b40 0x7fa39c0e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fa3a9564640 1 --2- 172.21.15.46:0/1289483691 >> v2:172.21.15.121:3300/0 conn(0x7fa39c0343d0 0x7fa39c053600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fdc89885640 1 -- 172.21.15.46:0/2273694670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c0e3150 msgr2=0x7fdc7c03e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fdc89885640 1 --2- 172.21.15.46:0/2273694670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c0e3150 0x7fdc7c03e7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c010180 tx=0x7fdc7802b910 comp rx=0 tx=0).stop 2023-12-16T21:13:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7f9ef5872640 1 Processor -- start 2023-12-16T21:13:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7f9ef5872640 1 -- start start 2023-12-16T21:13:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fa3aa566640 1 -- 172.21.15.46:0/1289483691 >> v2:172.21.15.121:3300/0 conn(0x7fa39c0343d0 msgr2=0x7fa39c053600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7fa3aa566640 1 --2- 172.21.15.46:0/1289483691 >> v2:172.21.15.121:3300/0 conn(0x7fa39c0343d0 0x7fa39c053600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7fa3aa566640 1 -- 172.21.15.46:0/1289483691 >> v1:172.21.15.46:6789/0 conn(0x7fa39c0e76b0 legacy=0x7fa39c052ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f90d68e5640 1 -- start start 2023-12-16T21:13:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7fa3aa566640 1 -- 172.21.15.46:0/1289483691 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0542f0 con 0x7fa39c053b40 2023-12-16T21:13:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7fa3a9564640 1 --2- 172.21.15.46:0/1289483691 >> v2:172.21.15.121:3300/0 conn(0x7fa39c0343d0 0x7fa39c053600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f71f5397640 1 -- 172.21.15.46:0/3222447739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f00b2060 msgr2=0x7f71f00b24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f71f5397640 1 --2- 172.21.15.46:0/3222447739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f00b2060 0x7f71f00b24d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f71f015c230 tx=0x7f71e402b930 comp rx=0 tx=0).stop 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f71f5397640 1 -- 172.21.15.46:0/3222447739 shutdown_connections 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f71f5397640 1 --2- 172.21.15.46:0/3222447739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f00b2060 0x7f71f00b24d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f71f5397640 1 --2- 172.21.15.46:0/3222447739 >> v2:172.21.15.121:3300/0 conn(0x7f71f00b0cf0 0x7f71f00b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f71f5397640 1 -- 172.21.15.46:0/3222447739 >> 172.21.15.46:0/3222447739 conn(0x7f71f006a4d0 msgr2=0x7f71f006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4489f25640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44840b2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4489f25640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44840b14c0 0x7f4484159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4489f25640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44840b2b70 con 0x7f44840ce600 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4489f25640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f448415de90 con 0x7f44840b14c0 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4489f25640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448415d280 con 0x7f44840b4cb0 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4483fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44840b14c0 0x7f4484159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4483fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44840b14c0 0x7f4484159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36276/0 (socket says 172.21.15.46:36276) 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f4483fff640 1 -- 172.21.15.46:0/1533484673 learned_addr learned my addr 172.21.15.46:0/1533484673 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f44837fe640 1 --2- 172.21.15.46:0/1533484673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44840b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.874+0000 7fcb34494640 1 -- start start 2023-12-16T21:13:45.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f44837fe640 1 -- 172.21.15.46:0/1533484673 >> v2:172.21.15.46:3300/0 conn(0x7f44840b14c0 msgr2=0x7f4484159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7fcb34494640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 0x7fcb2c1640c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7fcb34494640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcb2c164600 0x7fcb2c164ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb34494640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2c162db0 con 0x7fcb2c0b4c40 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb34494640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb2c0b5020 con 0x7fcb2c164600 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb34494640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c164ff0 con 0x7fcb2c1659a0 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f44837fe640 1 --2- 172.21.15.46:0/1533484673 >> v2:172.21.15.46:3300/0 conn(0x7f44840b14c0 0x7f4484159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f44837fe640 1 -- 172.21.15.46:0/1533484673 >> v1:172.21.15.121:6789/0 conn(0x7f44840ce600 legacy=0x7f44840b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f44827fc640 1 -- 172.21.15.46:0/1533484673 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582427352 0 0) 0x7f44840b2b70 con 0x7f44840ce600 2023-12-16T21:13:45.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f44837fe640 1 -- 172.21.15.46:0/1533484673 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448415d560 con 0x7f44840b4cb0 2023-12-16T21:13:45.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9ef5872640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9ef015d1e0 0x7f9ef00d3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f71f5397640 1 -- 172.21.15.46:0/3222447739 shutdown_connections 2023-12-16T21:13:45.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fa3aa566640 1 --2- 172.21.15.46:0/1289483691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c053b40 0x7fa39c0e7a90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00097a0 tx=0x7fa3a002b6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49c58d7dea83b2b1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 0x7fcb2c1640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 0x7fcb2c1640c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51148/0 (socket says 172.21.15.46:51148) 2023-12-16T21:13:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 -- 172.21.15.46:0/3150062316 learned_addr learned my addr 172.21.15.46:0/3150062316 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb32a31640 1 --2- 172.21.15.46:0/3150062316 >> v2:172.21.15.46:3300/0 conn(0x7fcb2c164600 0x7fcb2c164ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9ef5872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 0x7f9ef00cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9ef5872640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef00d0cc0 con 0x7f9ef00ce760 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9ef5872640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef00d04f0 con 0x7f9ef015d1e0 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9ef5872640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef00cef90 con 0x7f9ef00cfb00 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f71f5397640 1 -- 172.21.15.46:0/3222447739 wait complete. 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.873+0000 7f96fbfff640 1 -- 172.21.15.46:0/3071590618 wait complete. 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f96fbfff640 1 Processor -- start 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.875+0000 7f96fbfff640 1 -- start start 2023-12-16T21:13:45.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96fbfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 0x7f96f40382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96fbfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f404ea90 0x7f96f404a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96faffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 0x7f96f40382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96faffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 0x7f96f40382b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51174/0 (socket says 172.21.15.46:51174) 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96faffd640 1 -- 172.21.15.46:0/496179104 learned_addr learned my addr 172.21.15.46:0/496179104 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96fbfff640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96f40e1010 0x7f96f404a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f96fa7fc640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f404ea90 0x7f96f404a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9eeeffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9ef015d1e0 0x7f9ef00d3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9eef7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 0x7f9ef00cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fdc89885640 1 -- 172.21.15.46:0/2273694670 shutdown_connections 2023-12-16T21:13:45.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fdc89885640 1 --2- 172.21.15.46:0/2273694670 >> v2:172.21.15.46:3300/0 conn(0x7fdc7c0e7990 0x7fdc7c058450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fdc89885640 1 --2- 172.21.15.46:0/2273694670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c0e3150 0x7fdc7c03e7d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f9eef7fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 0x7f9ef00cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51164/0 (socket says 172.21.15.46:51164) 2023-12-16T21:13:45.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f9eef7fe640 1 -- 172.21.15.46:0/149768186 learned_addr learned my addr 172.21.15.46:0/149768186 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fdc89885640 1 -- 172.21.15.46:0/2273694670 >> 172.21.15.46:0/2273694670 conn(0x7fdc7c058eb0 msgr2=0x7fdc7c0592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fa3a8d63640 1 -- 172.21.15.46:0/1289483691 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0030710 con 0x7fa39c053b40 2023-12-16T21:13:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fdc89885640 1 -- 172.21.15.46:0/2273694670 shutdown_connections 2023-12-16T21:13:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f9701a4f640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96f40e1010 0x7f96f404a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f96fbfff640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f40d62e0 con 0x7f96f404ea90 2023-12-16T21:13:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f96fbfff640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f40d65c0 con 0x7f96f40e1010 2023-12-16T21:13:45.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f96fbfff640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f40d68a0 con 0x7f96f403a3f0 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d68e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d0142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d68e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 0x7f90d00cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d68e5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d0161720 0x7f90d00cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d68e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00ce340 con 0x7f90d00b3e40 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d68e5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00cc860 con 0x7f90d0161720 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d68e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d00ccb00 con 0x7f90d00b0cf0 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90cf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 0x7f90d00cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90cf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 0x7f90d00cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36282/0 (socket says 172.21.15.46:36282) 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90cf7fe640 1 -- 172.21.15.46:0/3273114862 learned_addr learned my addr 172.21.15.46:0/3273114862 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7f90d4e82640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d0161720 0x7f90d00cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f90cf7fe640 1 -- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 msgr2=0x7f90d0142980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-16T21:13:45.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f90cf7fe640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d0142980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f90cf7fe640 1 -- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d0161720 msgr2=0x7f90d00cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f90cf7fe640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d0161720 0x7f90d00cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f90cf7fe640 1 -- 172.21.15.46:0/3273114862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c8009a70 con 0x7f90d00b3e40 2023-12-16T21:13:45.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f90cf7fe640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 0x7f90d00cd8c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c4a0 tx=0x7f90c400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f44837fe640 1 --2- 172.21.15.46:0/1533484673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44840b2630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f447c009d90 tx=0x7f447c02e0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b5d41e941c99d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fa3a8d63640 1 -- 172.21.15.46:0/1289483691 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a002e050 con 0x7fa39c053b40 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f9eef7fe640 1 -- 172.21.15.46:0/149768186 >> v2:172.21.15.121:3300/0 conn(0x7f9ef015d1e0 msgr2=0x7f9ef00d3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f9eef7fe640 1 --2- 172.21.15.46:0/149768186 >> v2:172.21.15.121:3300/0 conn(0x7f9ef015d1e0 0x7f9ef00d3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f9eef7fe640 1 -- 172.21.15.46:0/149768186 >> v1:172.21.15.46:6789/0 conn(0x7f9ef00ce760 legacy=0x7f9ef00d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f9eef7fe640 1 -- 172.21.15.46:0/149768186 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef00d0f80 con 0x7f9ef00cfb00 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 -- 172.21.15.46:0/3150062316 >> v2:172.21.15.46:3300/0 conn(0x7fcb2c164600 msgr2=0x7fcb2c164ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 --2- 172.21.15.46:0/3150062316 >> v2:172.21.15.46:3300/0 conn(0x7fcb2c164600 0x7fcb2c164ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 -- 172.21.15.46:0/3150062316 >> v1:172.21.15.121:6789/0 conn(0x7fcb2c0b4c40 legacy=0x7fcb2c165290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.876+0000 7fcb31a2f640 1 -- 172.21.15.46:0/3150062316 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c163020 con 0x7fcb2c1659a0 2023-12-16T21:13:45.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f44827fc640 1 -- 172.21.15.46:0/1533484673 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c02ee60 con 0x7f44840b4cb0 2023-12-16T21:13:45.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f44827fc640 1 -- 172.21.15.46:0/1533484673 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c036440 con 0x7f44840b4cb0 2023-12-16T21:13:45.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7f44827fc640 1 -- 172.21.15.46:0/1533484673 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c0367d0 con 0x7f44840b4cb0 2023-12-16T21:13:45.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 -- 172.21.15.46:0/1289483691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c053b40 msgr2=0x7fa39c0e7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 --2- 172.21.15.46:0/1289483691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c053b40 0x7fa39c0e7a90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00097a0 tx=0x7fa3a002b6f0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fcb31a2f640 1 --2- 172.21.15.46:0/3150062316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 0x7fcb2c1640c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009e50 tx=0x7fcb2402bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf495e225663aee0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fa7fc640 1 -- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 msgr2=0x7f96f40382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fa7fc640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 0x7f96f40382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fa7fc640 1 -- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96f40e1010 msgr2=0x7f96f404a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fa7fc640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96f40e1010 0x7f96f404a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fa7fc640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec009400 con 0x7f96f404ea90 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f9701a4f640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96f40e1010 0x7f96f404a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96faffd640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 0x7f96f40382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:45.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fa7fc640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f404ea90 0x7f96f404a420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96f000b3f0 tx=0x7f96f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.877+0000 7fdc89885640 1 -- 172.21.15.46:0/2273694670 wait complete. 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 -- 172.21.15.46:0/1289483691 shutdown_connections 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 --2- 172.21.15.46:0/1289483691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c053b40 0x7fa39c0e7a90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 --2- 172.21.15.46:0/1289483691 >> v2:172.21.15.121:3300/0 conn(0x7fa39c0343d0 0x7fa39c053600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 -- 172.21.15.46:0/1289483691 >> 172.21.15.46:0/1289483691 conn(0x7fa39c055320 msgr2=0x7fa39c055730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4005ce0 con 0x7f90d00b3e40 2023-12-16T21:13:45.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c4005e40 con 0x7f90d00b3e40 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fcb3122e640 1 -- 172.21.15.46:0/3150062316 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2402cb20 con 0x7fcb2c1659a0 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fcb3122e640 1 -- 172.21.15.46:0/3150062316 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2402cc80 con 0x7fcb2c1659a0 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fcb3122e640 1 -- 172.21.15.46:0/3150062316 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24034610 con 0x7fcb2c1659a0 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c4007270 con 0x7f90d00b3e40 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f90d68e5640 1 -- 172.21.15.46:0/3273114862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d00ccde0 con 0x7f90d00b3e40 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f90d68e5640 1 -- 172.21.15.46:0/3273114862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d00cd3a0 con 0x7f90d00b3e40 2023-12-16T21:13:45.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 -- 172.21.15.46:0/1289483691 shutdown_connections 2023-12-16T21:13:45.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7fdc89885640 1 Processor -- start 2023-12-16T21:13:45.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f9eef7fe640 1 --2- 172.21.15.46:0/149768186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 0x7f9ef00cffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009e50 tx=0x7f9ee802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49ea54c2475168d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7fdc89885640 1 -- start start 2023-12-16T21:13:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f71f5397640 1 Processor -- start 2023-12-16T21:13:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f71f5397640 1 -- start start 2023-12-16T21:13:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f9eee7fc640 1 -- 172.21.15.46:0/149768186 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee802cb00 con 0x7f9ef00cfb00 2023-12-16T21:13:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f9eee7fc640 1 -- 172.21.15.46:0/149768186 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee802cc60 con 0x7f9ef00cfb00 2023-12-16T21:13:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f9eee7fc640 1 -- 172.21.15.46:0/149768186 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee80343f0 con 0x7f9ef00cfb00 2023-12-16T21:13:45.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7fa3aad67640 1 -- 172.21.15.46:0/1289483691 wait complete. 2023-12-16T21:13:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7fcb34494640 1 -- 172.21.15.46:0/3150062316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 msgr2=0x7fcb2c1640c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7fcb34494640 1 --2- 172.21.15.46:0/3150062316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 0x7fcb2c1640c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009e50 tx=0x7fcb2402bf80 comp rx=0 tx=0).stop 2023-12-16T21:13:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fcb34494640 1 -- 172.21.15.46:0/3150062316 shutdown_connections 2023-12-16T21:13:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fcb34494640 1 --2- 172.21.15.46:0/3150062316 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c1659a0 0x7fcb2c1640c0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb24009e50 tx=0x7fcb2402bf80 comp rx=0 tx=0).stop 2023-12-16T21:13:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fcb34494640 1 --2- 172.21.15.46:0/3150062316 >> v2:172.21.15.46:3300/0 conn(0x7fcb2c164600 0x7fcb2c164ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fcb34494640 1 -- 172.21.15.46:0/3150062316 >> 172.21.15.46:0/3150062316 conn(0x7fcb2c070900 msgr2=0x7fcb2c070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0011830 con 0x7f96f404ea90 2023-12-16T21:13:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f0011990 con 0x7f96f404ea90 2023-12-16T21:13:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0011d40 con 0x7f96f404ea90 2023-12-16T21:13:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f96fbfff640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f40d6b80 con 0x7f96f404ea90 2023-12-16T21:13:45.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f4489f25640 1 -- 172.21.15.46:0/1533484673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 msgr2=0x7f44840b2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.878+0000 7f4489f25640 1 --2- 172.21.15.46:0/1533484673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44840b2630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f447c009d90 tx=0x7f447c02e0b0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f4489f25640 1 -- 172.21.15.46:0/1533484673 shutdown_connections 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f4489f25640 1 --2- 172.21.15.46:0/1533484673 >> v2:172.21.15.46:3300/0 conn(0x7f44840b14c0 0x7f4484159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f4489f25640 1 --2- 172.21.15.46:0/1533484673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44840b2630 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f4489f25640 1 -- 172.21.15.46:0/1533484673 >> 172.21.15.46:0/1533484673 conn(0x7f448406a4d0 msgr2=0x7f448406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f4489f25640 1 -- 172.21.15.46:0/1533484673 shutdown_connections 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.879+0000 7f4489f25640 1 -- 172.21.15.46:0/1533484673 wait complete. 2023-12-16T21:13:45.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fa3aad67640 1 Processor -- start 2023-12-16T21:13:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fdc89885640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c058990 0x7fdc7c0d6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fdc89885640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc7c0e3150 0x7fdc7c006c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7f9ef5872640 1 -- 172.21.15.46:0/149768186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 msgr2=0x7f9ef00cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7f9ef5872640 1 --2- 172.21.15.46:0/149768186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 0x7f9ef00cffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009e50 tx=0x7f9ee802bf60 comp rx=0 tx=0).stop 2023-12-16T21:13:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fdc89885640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 0x7fdc7c007160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fdc89885640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c0d7670 con 0x7fdc7c0e3150 2023-12-16T21:13:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fdc89885640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c03a530 con 0x7fdc7c058990 2023-12-16T21:13:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fdc89885640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c039490 con 0x7fdc7c0e7990 2023-12-16T21:13:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f9ef5872640 1 -- 172.21.15.46:0/149768186 shutdown_connections 2023-12-16T21:13:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f9ef5872640 1 --2- 172.21.15.46:0/149768186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00cfb00 0x7f9ef00cffb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f9ef5872640 1 --2- 172.21.15.46:0/149768186 >> v2:172.21.15.121:3300/0 conn(0x7f9ef015d1e0 0x7f9ef00d3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f9ef5872640 1 -- 172.21.15.46:0/149768186 >> 172.21.15.46:0/149768186 conn(0x7f9ef009c300 msgr2=0x7f9ef00d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71f5397640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 0x7f71f015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 0x7fdc7c007160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc88883640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c058990 0x7fdc7c0d6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71f5397640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71f00b2060 0x7f71f014f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 0x7fdc7c007160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51192/0 (socket says 172.21.15.46:51192) 2023-12-16T21:13:45.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 -- 172.21.15.46:0/2405192936 learned_addr learned my addr 172.21.15.46:0/2405192936 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fcb34494640 1 -- 172.21.15.46:0/3150062316 shutdown_connections 2023-12-16T21:13:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f90d68e5640 1 -- 172.21.15.46:0/3273114862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0005250 con 0x7f90d00b3e40 2023-12-16T21:13:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71eeffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 0x7f71f015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71eeffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 0x7f71f015b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58388/0 (socket says 172.21.15.46:58388) 2023-12-16T21:13:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71eeffd640 1 -- 172.21.15.46:0/35406594 learned_addr learned my addr 172.21.15.46:0/35406594 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f0158690 0x7f71f0151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f9ef5872640 1 -- 172.21.15.46:0/149768186 shutdown_connections 2023-12-16T21:13:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f9ef5872640 1 -- 172.21.15.46:0/149768186 wait complete. 2023-12-16T21:13:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f4489f25640 1 Processor -- start 2023-12-16T21:13:45.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f4489f25640 1 -- start start 2023-12-16T21:13:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0164d90 con 0x7f71f0158690 2023-12-16T21:13:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01641a0 con 0x7f71f00b0cf0 2023-12-16T21:13:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01505f0 con 0x7f71f00b2060 2023-12-16T21:13:45.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7f71ee7fc640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71f00b2060 0x7f71f014f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 -- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c058990 msgr2=0x7fdc7c0d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c058990 0x7fdc7c0d6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 -- 172.21.15.46:0/2405192936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc7c0e3150 msgr2=0x7fdc7c006c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc7c0e3150 0x7fdc7c006c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fdc89084640 1 -- 172.21.15.46:0/2405192936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc780094d0 con 0x7fdc7c0e7990 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.880+0000 7fa3aad67640 1 -- start start 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fa3aad67640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 0x7fa39c0db820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fa3aad67640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa39c053b40 0x7fa39c047900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fa3aad67640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa39c0e76b0 0x7fa39c04e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fa3aad67640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c045440 con 0x7fa39c053b40 2023-12-16T21:13:45.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3aad67640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0dbd60 con 0x7fa39c0e76b0 2023-12-16T21:13:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3aad67640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1813e0 con 0x7fa39c0343d0 2023-12-16T21:13:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71eeffd640 1 -- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71f00b2060 msgr2=0x7f71f014f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71eeffd640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71f00b2060 0x7f71f014f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71eeffd640 1 -- 172.21.15.46:0/35406594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f0158690 msgr2=0x7f71f0151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71eeffd640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f0158690 0x7f71f0151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71eeffd640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e40094d0 con 0x7f71f00b0cf0 2023-12-16T21:13:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 0x7fa39c0db820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 0x7fa39c0db820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51222/0 (socket says 172.21.15.46:51222) 2023-12-16T21:13:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 -- 172.21.15.46:0/2774731468 learned_addr learned my addr 172.21.15.46:0/2774731468 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fdc89084640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 0x7fdc7c007160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc840681a0 tx=0x7fdc84067f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f90c40073d0 con 0x7f90d00b3e40 2023-12-16T21:13:45.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3aa566640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa39c0e76b0 0x7fa39c04e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8406bbb0 con 0x7fdc7c0e7990 2023-12-16T21:13:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8406bd10 con 0x7fdc7c0e7990 2023-12-16T21:13:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc84072400 con 0x7fdc7c0e7990 2023-12-16T21:13:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f4489f25640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 0x7f4484156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f4489f25640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44841ea950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f4489f25640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44840ce600 0x7f4484154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 -- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa39c0e76b0 msgr2=0x7fa39c04e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa39c0e76b0 0x7fa39c04e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 -- 172.21.15.46:0/2774731468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa39c053b40 msgr2=0x7fa39c047900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa39c053b40 0x7fa39c047900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fa3a9d65640 1 -- 172.21.15.46:0/2774731468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394009400 con 0x7fa39c0343d0 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f4489f25640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4484156360 con 0x7f44840ce600 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f4489f25640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44841556e0 con 0x7f44840b14c0 2023-12-16T21:13:45.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f4489f25640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44841559c0 con 0x7f44840b4cb0 2023-12-16T21:13:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7fa3a9d65640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 0x7fa39c0db820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4050140 tx=0x7fa3a4067f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 0x7f4484156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 0x7f4484156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58400/0 (socket says 172.21.15.46:58400) 2023-12-16T21:13:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f4482ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44841ea950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.921 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 -- 172.21.15.46:0/191620700 learned_addr learned my addr 172.21.15.46:0/191620700 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7fa3aa566640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa39c0e76b0 0x7fa39c04e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71eeffd640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 0x7f71f015b4e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f71e800ba50 tx=0x7f71e800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e800cc10 con 0x7f71f00b0cf0 2023-12-16T21:13:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e800cd70 con 0x7f71f00b0cf0 2023-12-16T21:13:45.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e8014680 con 0x7f71f00b0cf0 2023-12-16T21:13:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.881+0000 7fcb34494640 1 -- 172.21.15.46:0/3150062316 wait complete. 2023-12-16T21:13:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7fcb34494640 1 Processor -- start 2023-12-16T21:13:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f015ba20 con 0x7f71f00b0cf0 2023-12-16T21:13:45.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7fcb34494640 1 -- start start 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c039770 con 0x7fdc7c0e7990 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.882+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c039d30 con 0x7fdc7c0e7990 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f01646d0 con 0x7f71f00b0cf0 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 -- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 msgr2=0x7f44841ea950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44841ea950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 -- 172.21.15.46:0/191620700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44840ce600 msgr2=0x7f4484154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44840ce600 0x7f4484154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7f44837fe640 1 -- 172.21.15.46:0/191620700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c009a70 con 0x7f44840b14c0 2023-12-16T21:13:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7fcb34494640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb2c0b4c40 0x7fcb2c14a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb34494640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 0x7fcb2c15d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb34494640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb2c15d8c0 0x7fcb2c15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb34494640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c15fc50 con 0x7fcb2c0b4c40 2023-12-16T21:13:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb31a2f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 0x7fcb2c15d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7f44837fe640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 0x7f4484156780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f447c009d90 tx=0x7f447c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a406bea0 con 0x7fa39c0343d0 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.883+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a4067850 con 0x7fa39c0343d0 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4072400 con 0x7fa39c0343d0 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0009400 con 0x7fa39c0343d0 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c046e20 con 0x7fa39c0343d0 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7f9ef5872640 1 Processor -- start 2023-12-16T21:13:45.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7f9ef5872640 1 -- start start 2023-12-16T21:13:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb31a2f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 0x7fcb2c15d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51238/0 (socket says 172.21.15.46:51238) 2023-12-16T21:13:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb31a2f640 1 -- 172.21.15.46:0/2514209431 learned_addr learned my addr 172.21.15.46:0/2514209431 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c15ff30 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c160210 con 0x7fcb2c164600 2023-12-16T21:13:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fcb32a31640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb2c15d8c0 0x7fcb2c15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f9ef5872640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 0x7f9ef00cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c037ea0 con 0x7f44840b14c0 2023-12-16T21:13:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c03f830 con 0x7f44840b14c0 2023-12-16T21:13:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c03fbc0 con 0x7f44840b14c0 2023-12-16T21:13:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c009250 con 0x7fa39c0343d0 2023-12-16T21:13:45.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f9ef5872640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ef00cfb00 0x7f9ef00b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c0057b0 con 0x7fdc7c0e7990 2023-12-16T21:13:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f9eeeffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 0x7f9ef00cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.884+0000 7f96fbfff640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f40d2c30 con 0x7f96f404ea90 2023-12-16T21:13:45.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4484155ca0 con 0x7f44840b14c0 2023-12-16T21:13:45.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44841e8a80 con 0x7f44840b14c0 2023-12-16T21:13:45.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f0069400 con 0x7f71f00b0cf0 2023-12-16T21:13:45.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb32a31640 1 -- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 msgr2=0x7fcb2c15d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f9eeeffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 0x7f9ef00cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51246/0 (socket says 172.21.15.46:51246) 2023-12-16T21:13:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb32a31640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 0x7fcb2c15d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb32a31640 1 -- 172.21.15.46:0/2514209431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb2c0b4c40 msgr2=0x7fcb2c14a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f40d70a0 con 0x7f96f404ea90 2023-12-16T21:13:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb32a31640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb2c0b4c40 0x7fcb2c14a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.885+0000 7f9ef5872640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ef015d1e0 0x7f9ef01f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb32a31640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb24009b30 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb31a2f640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 0x7fcb2c15d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdc84072560 con 0x7fdc7c0e7990 2023-12-16T21:13:45.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f9eeeffd640 1 -- 172.21.15.46:0/3514674313 learned_addr learned my addr 172.21.15.46:0/3514674313 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef00ccb50 con 0x7f9ef00cfb00 2023-12-16T21:13:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef00cbf40 con 0x7f9ef015d1e0 2023-12-16T21:13:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fcb32a31640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb2c15d8c0 0x7fcb2c15c260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00d490 tx=0x7fcb1c00d950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fdc817fa640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc6806ae40 0x7fdc6806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f9eef7fe640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ef015d1e0 0x7f9ef01f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef00cc220 con 0x7f9ef00ce760 2023-12-16T21:13:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc84103480 con 0x7fdc7c0e7990 2023-12-16T21:13:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f71e8007370 con 0x7f71f00b0cf0 2023-12-16T21:13:45.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7fdc88883640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc6806ae40 0x7fdc6806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f90ccff9640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90bc06ae90 0x7f90bc06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90c40886c0 con 0x7f90d00b3e40 2023-12-16T21:13:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f71d77fe640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71dc06af10 0x7f71dc06d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.886+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71e8089680 con 0x7f71f00b0cf0 2023-12-16T21:13:45.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9ee7fff640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ef00cfb00 0x7f9ef00b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f90cffff640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90bc06ae90 0x7f90bc06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9eeeffd640 1 -- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ef015d1e0 msgr2=0x7f9ef01f1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9eeeffd640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ef015d1e0 0x7f9ef01f1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9eeeffd640 1 -- 172.21.15.46:0/3514674313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ef00cfb00 msgr2=0x7f9ef00b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9eeeffd640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ef00cfb00 0x7f9ef00b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9eeeffd640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee8009b30 con 0x7f9ef00ce760 2023-12-16T21:13:45.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f96f001a400 con 0x7f96f404ea90 2023-12-16T21:13:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f71ee7fc640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71dc06af10 0x7f71dc06d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.887+0000 7f9eeeffd640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 0x7f9ef00cab90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ee000e870 tx=0x7f9ee000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f96e3fff640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96e806ae70 0x7f96e806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7fdc88883640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc6806ae40 0x7fdc6806d300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc78036000 tx=0x7fdc7803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f447c03d070 con 0x7f44840b14c0 2023-12-16T21:13:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa3a4072620 con 0x7fa39c0343d0 2023-12-16T21:13:45.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f90cffff640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90bc06ae90 0x7f90bc06d350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90c000b7f0 tx=0x7f90c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f71ee7fc640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71dc06af10 0x7f71dc06d3d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f71f0159090 tx=0x7f71e00090e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44841564a0 con 0x7f44840b14c0 2023-12-16T21:13:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f96faffd640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96e806ae70 0x7f96e806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.939 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7fa39a7fc640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa38806ae90 0x7fa38806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f446bfff640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f447006ae90 0x7f447006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3a41033e0 con 0x7fa39c0343d0 2023-12-16T21:13:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7fa3a9564640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa38806ae90 0x7fa38806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f447c0ac3b0 con 0x7f44840b14c0 2023-12-16T21:13:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f96faffd640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96e806ae70 0x7f96e806d330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f96ec02b750 tx=0x7f96ec008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.888+0000 7f4482ffd640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f447006ae90 0x7f447006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.889+0000 7fa3a9564640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa38806ae90 0x7fa38806d350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa39c03abe0 tx=0x7fa394009630 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.889+0000 7f4482ffd640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f447006ae90 0x7f447006d350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4484202bb0 tx=0x7f4478009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.890+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96f0058b90 con 0x7f96f404ea90 2023-12-16T21:13:45.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.890+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f90c40538e0 con 0x7f90d00b3e40 2023-12-16T21:13:45.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.891+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee00094f0 con 0x7f9ef00ce760 2023-12-16T21:13:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef00cc500 con 0x7f9ef00ce760 2023-12-16T21:13:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef00cb0d0 con 0x7f9ef00ce760 2023-12-16T21:13:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdc840ce490 con 0x7fdc7c0e7990 2023-12-16T21:13:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c018070 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f71e8054730 con 0x7f71f00b0cf0 2023-12-16T21:13:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2c1604f0 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.892+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2c15dd70 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.893+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef0067420 con 0x7f9ef00ce760 2023-12-16T21:13:45.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.893+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb2c067420 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.895+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f96f40d70a0 con 0x7f96f404ea90 2023-12-16T21:13:45.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.896+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee0005990 con 0x7f9ef00ce760 2023-12-16T21:13:45.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.896+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0005d00 con 0x7f9ef00ce760 2023-12-16T21:13:45.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.896+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa3a40ce470 con 0x7fa39c0343d0 2023-12-16T21:13:45.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.896+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c004020 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.897+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f447c0774d0 con 0x7f44840b14c0 2023-12-16T21:13:45.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.898+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c004190 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.898+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fcb1c0043b0 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.898+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9ee000c940 con 0x7f9ef00ce760 2023-12-16T21:13:45.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.899+0000 7fcb22ffd640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcb1006ae40 0x7fcb1006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.899+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb1c089de0 con 0x7fcb2c15d8c0 2023-12-16T21:13:45.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.899+0000 7f9eecff9640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ed406ae40 0x7f9ed406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:45.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.899+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ee0089d20 con 0x7f9ef00ce760 2023-12-16T21:13:45.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.899+0000 7fcb32230640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcb1006ae40 0x7fcb1006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.900+0000 7f9ee7fff640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ed406ae40 0x7f9ed406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:45.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.900+0000 7fcb32230640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcb1006ae40 0x7fcb1006d300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcb28000f60 tx=0x7fcb280079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.900+0000 7f9ee7fff640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ed406ae40 0x7f9ed406d300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00c9c70 tx=0x7f9ed8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:45.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.902+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9ee0054ef0 con 0x7f9ef00ce760 2023-12-16T21:13:45.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:45.902+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcb1c054ef0 con 0x7fcb2c15d8c0 2023-12-16T21:13:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.296+0000 7f90d68e5640 1 -- 172.21.15.46:0/3273114862 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f90b00057d0 con 0x7f90d00b3e40 2023-12-16T21:13:46.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.297+0000 7f90ccff9640 1 -- 172.21.15.46:0/3273114862 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f90c4056f00 con 0x7f90d00b3e40 2023-12-16T21:13:46.297 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.300+0000 7f90b67fc640 1 -- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90bc06ae90 msgr2=0x7f90bc06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.300+0000 7f90b67fc640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90bc06ae90 0x7f90bc06d350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90c000b7f0 tx=0x7f90c0008040 comp rx=0 tx=0).stop 2023-12-16T21:13:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.300+0000 7f90b67fc640 1 -- 172.21.15.46:0/3273114862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 msgr2=0x7f90d00cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.300+0000 7f90b67fc640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 0x7f90d00cd8c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c4a0 tx=0x7f90c400c960 comp rx=0 tx=0).stop 2023-12-16T21:13:46.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.300+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fdc7c056f90 con 0x7fdc7c0e7990 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 -- 172.21.15.46:0/3273114862 shutdown_connections 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90bc06ae90 0x7f90bc06d350 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90d00b3e40 0x7f90d00cd8c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d0161720 0x7f90d00cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 --2- 172.21.15.46:0/3273114862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90d00b0cf0 0x7f90d0142980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 -- 172.21.15.46:0/3273114862 >> 172.21.15.46:0/3273114862 conn(0x7f90d006a4d0 msgr2=0x7f90d015f360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 -- 172.21.15.46:0/3273114862 shutdown_connections 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7f90b67fc640 1 -- 172.21.15.46:0/3273114862 wait complete. 2023-12-16T21:13:46.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.301+0000 7fdc817fa640 1 -- 172.21.15.46:0/2405192936 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc840d1ab0 con 0x7fdc7c0e7990 2023-12-16T21:13:46.303 INFO:teuthology.orchestra.run.smithi046.stdout:55834574853 2023-12-16T21:13:46.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.304+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f71f00b25e0 con 0x7f71f00b0cf0 2023-12-16T21:13:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.305+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc6806ae40 msgr2=0x7fdc6806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.305+0000 7fdc89885640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc6806ae40 0x7fdc6806d300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdc78036000 tx=0x7fdc7803c040 comp rx=0 tx=0).stop 2023-12-16T21:13:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.305+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 msgr2=0x7fdc7c007160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.305+0000 7fdc89885640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 0x7fdc7c007160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdc840681a0 tx=0x7fdc84067f00 comp rx=0 tx=0).stop 2023-12-16T21:13:46.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.305+0000 7f71d77fe640 1 -- 172.21.15.46:0/35406594 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f71e8019090 con 0x7f71f00b0cf0 2023-12-16T21:13:46.306 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 shutdown_connections 2023-12-16T21:13:46.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc6806ae40 0x7fdc6806d300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc7c0e3150 0x7fdc7c006c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc7c0e7990 0x7fdc7c007160 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 --2- 172.21.15.46:0/2405192936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc7c058990 0x7fdc7c0d6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 >> 172.21.15.46:0/2405192936 conn(0x7fdc7c058eb0 msgr2=0x7fdc7c03d440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 shutdown_connections 2023-12-16T21:13:46.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.306+0000 7fdc89885640 1 -- 172.21.15.46:0/2405192936 wait complete. 2023-12-16T21:13:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.308+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71dc06af10 msgr2=0x7f71dc06d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.308+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71dc06af10 0x7f71dc06d3d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f71f0159090 tx=0x7f71e00090e0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 msgr2=0x7f71f015b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 0x7f71f015b4e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f71e800ba50 tx=0x7f71e800bf10 comp rx=0 tx=0).stop 2023-12-16T21:13:46.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f96f40d70a0 con 0x7f96f404ea90 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 shutdown_connections 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71dc06af10 0x7f71dc06d3d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71f0158690 0x7f71f0151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71f00b2060 0x7f71f014f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 --2- 172.21.15.46:0/35406594 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71f00b0cf0 0x7f71f015b4e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 >> 172.21.15.46:0/35406594 conn(0x7f71f006a4d0 msgr2=0x7f71f0155f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.309+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 shutdown_connections 2023-12-16T21:13:46.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.310+0000 7f71f5397640 1 -- 172.21.15.46:0/35406594 wait complete. 2023-12-16T21:13:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.310+0000 7f96e3fff640 1 -- 172.21.15.46:0/496179104 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f96f40d70a0 con 0x7f96f404ea90 2023-12-16T21:13:46.313 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:46.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.311+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa39c035bb0 con 0x7fa39c0343d0 2023-12-16T21:13:46.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.312+0000 7fa39a7fc640 1 -- 172.21.15.46:0/2774731468 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa3a40d1a90 con 0x7fa39c0343d0 2023-12-16T21:13:46.314 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:46.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.313+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96e806ae70 msgr2=0x7f96e806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.313+0000 7f96e1ffb640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96e806ae70 0x7f96e806d330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f96ec02b750 tx=0x7f96ec008030 comp rx=0 tx=0).stop 2023-12-16T21:13:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.313+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f404ea90 msgr2=0x7f96f404a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.313+0000 7f96e1ffb640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f404ea90 0x7f96f404a420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96f000b3f0 tx=0x7f96f000b8b0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f9ef5872640 1 -- 172.21.15.46:0/3514674313 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9ef00d0850 con 0x7f9ef00ce760 2023-12-16T21:13:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 shutdown_connections 2023-12-16T21:13:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96e806ae70 0x7f96e806d330 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96f404ea90 0x7f96f404a420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96f40e1010 0x7f96f404a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 --2- 172.21.15.46:0/496179104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96f403a3f0 0x7f96f40382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4484069400 con 0x7f44840b14c0 2023-12-16T21:13:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 >> 172.21.15.46:0/496179104 conn(0x7f96f4054c70 msgr2=0x7f96f40e02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 shutdown_connections 2023-12-16T21:13:46.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f9eecff9640 1 -- 172.21.15.46:0/3514674313 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ee0021090 con 0x7f9ef00ce760 2023-12-16T21:13:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f96e1ffb640 1 -- 172.21.15.46:0/496179104 wait complete. 2023-12-16T21:13:46.318 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:46.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.314+0000 7f446bfff640 1 -- 172.21.15.46:0/191620700 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f447c07aaf0 con 0x7f44840b14c0 2023-12-16T21:13:46.318 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.316+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa38806ae90 msgr2=0x7fa38806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.316+0000 7fa3aad67640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa38806ae90 0x7fa38806d350 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa39c03abe0 tx=0x7fa394009630 comp rx=0 tx=0).stop 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.316+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 msgr2=0x7fa39c0db820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.316+0000 7fa3aad67640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 0x7fa39c0db820 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4050140 tx=0x7fa3a4067f00 comp rx=0 tx=0).stop 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.316+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 shutdown_connections 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.316+0000 7fa3aad67640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa38806ae90 0x7fa38806d350 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.317+0000 7fa3aad67640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa39c053b40 0x7fa39c047900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.317+0000 7fa3aad67640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa39c0e76b0 0x7fa39c04e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.317+0000 7fa3aad67640 1 --2- 172.21.15.46:0/2774731468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa39c0343d0 0x7fa39c0db820 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.317+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 >> 172.21.15.46:0/2774731468 conn(0x7fa39c055320 msgr2=0x7fa39c055800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.317+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 shutdown_connections 2023-12-16T21:13:46.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.317+0000 7fa3aad67640 1 -- 172.21.15.46:0/2774731468 wait complete. 2023-12-16T21:13:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fcb2c160070 con 0x7fcb2c15d8c0 2023-12-16T21:13:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f447006ae90 msgr2=0x7f447006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f447006ae90 0x7f447006d350 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4484202bb0 tx=0x7f4478009040 comp rx=0 tx=0).stop 2023-12-16T21:13:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 msgr2=0x7f4484156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 0x7f4484156780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f447c009d90 tx=0x7f447c004a40 comp rx=0 tx=0).stop 2023-12-16T21:13:46.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 shutdown_connections 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f447006ae90 0x7f447006d350 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44840ce600 0x7f4484154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44840b4cb0 0x7f44841ea950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 --2- 172.21.15.46:0/191620700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44840b14c0 0x7f4484156780 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 >> 172.21.15.46:0/191620700 conn(0x7f448406a4d0 msgr2=0x7f448415bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 shutdown_connections 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7f4489f25640 1 -- 172.21.15.46:0/191620700 wait complete. 2023-12-16T21:13:46.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.318+0000 7fcb22ffd640 1 -- 172.21.15.46:0/2514209431 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcb2c160070 con 0x7fcb2c15d8c0 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 -- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ed406ae40 msgr2=0x7f9ed406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ed406ae40 0x7f9ed406d300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00c9c70 tx=0x7f9ed8009040 comp rx=0 tx=0).stop 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 -- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 msgr2=0x7f9ef00cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 0x7f9ef00cab90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ee000e870 tx=0x7f9ee000ed30 comp rx=0 tx=0).stop 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 -- 172.21.15.46:0/3514674313 shutdown_connections 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ed406ae40 0x7f9ed406d300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ef00cfb00 0x7f9ef00b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ef015d1e0 0x7f9ef01f1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 --2- 172.21.15.46:0/3514674313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ef00ce760 0x7f9ef00cab90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 -- 172.21.15.46:0/3514674313 >> 172.21.15.46:0/3514674313 conn(0x7f9ef009c300 msgr2=0x7f9ef00733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 -- 172.21.15.46:0/3514674313 shutdown_connections 2023-12-16T21:13:46.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.319+0000 7f9ee5ffb640 1 -- 172.21.15.46:0/3514674313 wait complete. 2023-12-16T21:13:46.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2023-12-16T21:13:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcb1006ae40 msgr2=0x7fcb1006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcb1006ae40 0x7fcb1006d300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcb28000f60 tx=0x7fcb280079b0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb2c15d8c0 msgr2=0x7fcb2c15c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb2c15d8c0 0x7fcb2c15c260 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00d490 tx=0x7fcb1c00d950 comp rx=0 tx=0).stop 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 shutdown_connections 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcb1006ae40 0x7fcb1006d300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb2c15d8c0 0x7fcb2c15c260 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb2c164600 0x7fcb2c15d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 --2- 172.21.15.46:0/2514209431 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcb2c0b4c40 0x7fcb2c14a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.322+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 >> 172.21.15.46:0/2514209431 conn(0x7fcb2c070900 msgr2=0x7fcb2c0b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.323+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 shutdown_connections 2023-12-16T21:13:46.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:46.323+0000 7fcb34494640 1 -- 172.21.15.46:0/2514209431 wait complete. 2023-12-16T21:13:46.327 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2023-12-16T21:13:46.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2023-12-16T21:13:46.337 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2023-12-16T21:13:46.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2023-12-16T21:13:46.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-12-16T21:13:46.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.5 2023-12-16T21:13:46.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-12-16T21:13:47.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-16T21:13:47.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-16T21:13:47.335 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-16T21:13:47.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-16T21:13:47.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-16T21:13:47.340 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-16T21:13:47.340 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-16T21:13:47.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-16T21:13:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.504+0000 7f2d3ec02640 1 Processor -- start 2023-12-16T21:13:47.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.505+0000 7f2d3ec02640 1 -- start start 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.506+0000 7f2d3ec02640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d300e78f0 0x7f2d30035030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.506+0000 7f2d3ec02640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2d3003ac80 0x7f2d30048390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.506+0000 7f2d3ec02640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d30056330 con 0x7f2d300488d0 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.506+0000 7f2d3ec02640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d30048ce0 con 0x7f2d300e78f0 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.506+0000 7f2d3ec02640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d300e7cd0 con 0x7f2d3003ac80 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.507+0000 7f2d3e401640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f2d300488d0 0x7f2d300537c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53252/0 (socket says 172.21.15.46:53252) 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.507+0000 7f2d3e401640 1 -- 172.21.15.46:0/1114467487 learned_addr learned my addr 172.21.15.46:0/1114467487 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.507+0000 7f2d3d3ff640 1 --2- 172.21.15.46:0/1114467487 >> v2:172.21.15.121:3300/0 conn(0x7f2d3003ac80 0x7f2d30048390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.507+0000 7f2d3dc00640 1 --2- 172.21.15.46:0/1114467487 >> v2:172.21.15.46:3300/0 conn(0x7f2d300e78f0 0x7f2d30035030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.508+0000 7f2d3dc00640 1 -- 172.21.15.46:0/1114467487 >> v2:172.21.15.121:3300/0 conn(0x7f2d3003ac80 msgr2=0x7f2d30048390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.508+0000 7f2d3dc00640 1 --2- 172.21.15.46:0/1114467487 >> v2:172.21.15.121:3300/0 conn(0x7f2d3003ac80 0x7f2d30048390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.508+0000 7f2d3dc00640 1 -- 172.21.15.46:0/1114467487 >> v1:172.21.15.121:6789/0 conn(0x7f2d300488d0 legacy=0x7f2d300537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.508+0000 7f2d3dc00640 1 -- 172.21.15.46:0/1114467487 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30056050 con 0x7f2d300e78f0 2023-12-16T21:13:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.509+0000 7f2d3dc00640 1 --2- 172.21.15.46:0/1114467487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d300e78f0 0x7f2d30035030 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d30052f40 tx=0x7f2d2802b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d570b0965e3e08 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.509+0000 7f2d3cbfe640 1 -- 172.21.15.46:0/1114467487 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d28030710 con 0x7f2d300e78f0 2023-12-16T21:13:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.510+0000 7f2d3cbfe640 1 -- 172.21.15.46:0/1114467487 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2802e050 con 0x7f2d300e78f0 2023-12-16T21:13:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.511+0000 7f2d3ec02640 1 -- 172.21.15.46:0/1114467487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d300e78f0 msgr2=0x7f2d30035030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.511+0000 7f2d3ec02640 1 --2- 172.21.15.46:0/1114467487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d300e78f0 0x7f2d30035030 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d30052f40 tx=0x7f2d2802b680 comp rx=0 tx=0).stop 2023-12-16T21:13:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.511+0000 7f2d3ec02640 1 -- 172.21.15.46:0/1114467487 shutdown_connections 2023-12-16T21:13:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.511+0000 7f2d3ec02640 1 --2- 172.21.15.46:0/1114467487 >> v2:172.21.15.121:3300/0 conn(0x7f2d3003ac80 0x7f2d30048390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.511+0000 7f2d3ec02640 1 --2- 172.21.15.46:0/1114467487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d300e78f0 0x7f2d30035030 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.511+0000 7f2d3ec02640 1 -- 172.21.15.46:0/1114467487 >> 172.21.15.46:0/1114467487 conn(0x7f2d3001e690 msgr2=0x7f2d300553c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.512+0000 7f2d3ec02640 1 -- 172.21.15.46:0/1114467487 shutdown_connections 2023-12-16T21:13:47.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.512+0000 7f2d3ec02640 1 -- 172.21.15.46:0/1114467487 wait complete. 2023-12-16T21:13:47.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.514+0000 7f2d3ec02640 1 Processor -- start 2023-12-16T21:13:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.514+0000 7f2d3ec02640 1 -- start start 2023-12-16T21:13:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.515+0000 7f2d3ec02640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 0x7f2d3004b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.515+0000 7f2d3ec02640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d300488d0 0x7f2d300e8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 0x7f2d3004b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 0x7f2d3004b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36372/0 (socket says 172.21.15.46:36372) 2023-12-16T21:13:47.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3ec02640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d300e78f0 0x7f2d300cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3ec02640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300cee40 con 0x7f2d3003ac80 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3ec02640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3004b9e0 con 0x7f2d300488d0 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3ec02640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d300cdf70 con 0x7f2d300e78f0 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3d3ff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d300488d0 0x7f2d300e8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3d3ff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d300488d0 0x7f2d300e8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58428/0 (socket says 172.21.15.46:58428) 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3d3ff640 1 -- 172.21.15.46:0/2746342677 learned_addr learned my addr 172.21.15.46:0/2746342677 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.516+0000 7f2d3e401640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d300e78f0 0x7f2d300cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.517+0000 7f2d3e401640 1 -- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d300488d0 msgr2=0x7f2d300e8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.517+0000 7f2d3e401640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d300488d0 0x7f2d300e8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.517+0000 7f2d3e401640 1 -- 172.21.15.46:0/2746342677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 msgr2=0x7f2d3004b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.518+0000 7f2d3e401640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 0x7f2d3004b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.518+0000 7f2d3e401640 1 -- 172.21.15.46:0/2746342677 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d28009400 con 0x7f2d300e78f0 2023-12-16T21:13:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.518+0000 7f2d3dc00640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 0x7f2d3004b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.518+0000 7f2d3e401640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d300e78f0 0x7f2d300cf4d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2d38068f20 tx=0x7f2d38067f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.518+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3806bc90 con 0x7f2d300e78f0 2023-12-16T21:13:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.519+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3806be10 con 0x7f2d300e78f0 2023-12-16T21:13:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.519+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38072400 con 0x7f2d300e78f0 2023-12-16T21:13:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.519+0000 7f2d3ec02640 1 -- 172.21.15.46:0/2746342677 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d300ce1c0 con 0x7f2d300e78f0 2023-12-16T21:13:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.519+0000 7f2d3ec02640 1 -- 172.21.15.46:0/2746342677 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d300ce780 con 0x7f2d300e78f0 2023-12-16T21:13:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.522+0000 7f2d3ec02640 1 -- 172.21.15.46:0/2746342677 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d20005250 con 0x7f2d300e78f0 2023-12-16T21:13:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.523+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2d38072620 con 0x7f2d300e78f0 2023-12-16T21:13:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.524+0000 7f2d2e7fc640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2406b0b0 0x7f2d2406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.524+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d38103520 con 0x7f2d300e78f0 2023-12-16T21:13:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.524+0000 7f2d3dc00640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2406b0b0 0x7f2d2406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.525+0000 7f2d3dc00640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2406b0b0 0x7f2d2406d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2d28034000 tx=0x7f2d28008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.527+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2d380ce6f0 con 0x7f2d300e78f0 2023-12-16T21:13:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.559+0000 7f1fb8a56640 1 Processor -- start 2023-12-16T21:13:47.559 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.559+0000 7f1fb8a56640 1 -- start start 2023-12-16T21:13:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.560+0000 7f2db4b41640 1 Processor -- start 2023-12-16T21:13:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.560+0000 7f2db4b41640 1 -- start start 2023-12-16T21:13:47.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.560+0000 7f1fb8a56640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 0x7f1fb40b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb8a56640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb4071640 0x7f1fb40cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb8a56640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fb40b48a0 con 0x7f1fb40b4440 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb8a56640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fb4071ab0 con 0x7f1fb414b050 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 0x7f1fb40b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 0x7f1fb40b3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36376/0 (socket says 172.21.15.46:36376) 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb259c640 1 -- 172.21.15.46:0/445448365 learned_addr learned my addr 172.21.15.46:0/445448365 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f1fb8a56640 1 -- 172.21.15.46:0/445448365 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb4071cd0 con 0x7f1fb4071640 2023-12-16T21:13:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.561+0000 7f92c89b0640 1 Processor -- start 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f7c1b59e640 1 Processor -- start 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f7c1b59e640 1 -- start start 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f7c1b59e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c156dd0 0x7f7c1c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f7c1b59e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c1c071260 0x7f7c1c0716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1b59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c1c0b2170 con 0x7f7c1c1597f0 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb2d9d640 1 --2- 172.21.15.46:0/445448365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb4071640 0x7f1fb40cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb159a640 1 -- 172.21.15.46:0/445448365 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147353030 0 0) 0x7f1fb40b48a0 con 0x7f1fb40b4440 2023-12-16T21:13:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb159a640 1 -- 172.21.15.46:0/445448365 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f9c003520 con 0x7f1fb40b4440 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb2d9d640 1 -- 172.21.15.46:0/445448365 >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 msgr2=0x7f1fb40b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb2d9d640 1 --2- 172.21.15.46:0/445448365 >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 0x7f1fb40b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb2d9d640 1 -- 172.21.15.46:0/445448365 >> v1:172.21.15.46:6789/0 conn(0x7f1fb40b4440 legacy=0x7f1fb4070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f1fb2d9d640 1 -- 172.21.15.46:0/445448365 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb4071fb0 con 0x7f1fb4071640 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f1fb259c640 1 --2- 172.21.15.46:0/445448365 >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 0x7f1fb40b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2db4b41640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2da80e3090 0x7f2da8057d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2db4b41640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2da803e810 0x7f2da8053bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2db4b41640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da8054100 0x7f2da80e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2db4b41640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da8006170 con 0x7f2da803e810 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2db4b41640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2da8054570 con 0x7f2da80e3090 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2db4b41640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80582a0 con 0x7f2da8054100 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f2daed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da8054100 0x7f2da80e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2daed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da8054100 0x7f2da80e6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51260/0 (socket says 172.21.15.46:51260) 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2daed9d640 1 -- 172.21.15.46:0/425470915 learned_addr learned my addr 172.21.15.46:0/425470915 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dae59c640 1 --2- 172.21.15.46:0/425470915 >> v2:172.21.15.121:3300/0 conn(0x7f2da80e3090 0x7f2da8057d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dadd9b640 1 --2- 172.21.15.46:0/425470915 >> v2:172.21.15.46:3300/0 conn(0x7f2da803e810 0x7f2da8053bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dae59c640 1 -- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da8054100 msgr2=0x7f2da80e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dae59c640 1 --2- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da8054100 0x7f2da80e6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dae59c640 1 -- 172.21.15.46:0/425470915 >> v2:172.21.15.46:3300/0 conn(0x7f2da803e810 msgr2=0x7f2da8053bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dae59c640 1 --2- 172.21.15.46:0/425470915 >> v2:172.21.15.46:3300/0 conn(0x7f2da803e810 0x7f2da8053bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f2dae59c640 1 -- 172.21.15.46:0/425470915 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da8058540 con 0x7f2da80e3090 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f17942ad640 1 Processor -- start 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f17942ad640 1 -- start start 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f17942ad640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f178c0b1880 0x7f178c0b4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f1792049640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f178c159210 0x7f178c15a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53542/0 (socket says 172.21.15.46:53542) 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f1792049640 1 -- 172.21.15.46:0/111562721 learned_addr learned my addr 172.21.15.46:0/111562721 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c0cd020 con 0x7f178c159210 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f179284a640 1 --2- 172.21.15.46:0/111562721 >> v2:172.21.15.46:3300/0 conn(0x7f178c0b1880 0x7f178c0b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c0cd300 con 0x7f178c0b40b0 2023-12-16T21:13:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178c0b4a70 con 0x7f178c0b1880 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7fcf7beda640 1 Processor -- start 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7fcf7beda640 1 -- start start 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf7beda640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcf74159210 0x7fcf7415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf7beda640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7415a830 con 0x7fcf740b40b0 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf7beda640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7415dc80 con 0x7fcf740b34a0 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf7beda640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf740cd020 con 0x7fcf74159210 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.562+0000 7f92c89b0640 1 -- start start 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f92c89b0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f92c4164e80 0x7f92c406c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f92c89b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 0x7f92c409b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f92c89b0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92c40d19e0 con 0x7f92c40b5f20 2023-12-16T21:13:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f92c89b0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92c40b7230 con 0x7f92c4164e80 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.563+0000 7f92c89b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40b6f50 con 0x7f92c409aef0 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 0x7f92c409b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 0x7f92c409b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51284/0 (socket says 172.21.15.46:51284) 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 -- 172.21.15.46:0/4165436310 learned_addr learned my addr 172.21.15.46:0/4165436310 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c259c640 1 --2- 172.21.15.46:0/4165436310 >> v2:172.21.15.46:3300/0 conn(0x7f92c4164e80 0x7f92c406c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 -- 172.21.15.46:0/4165436310 >> v2:172.21.15.46:3300/0 conn(0x7f92c4164e80 msgr2=0x7f92c406c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 --2- 172.21.15.46:0/4165436310 >> v2:172.21.15.46:3300/0 conn(0x7f92c4164e80 0x7f92c406c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 -- 172.21.15.46:0/4165436310 >> v1:172.21.15.46:6789/0 conn(0x7f92c40b5f20 legacy=0x7f92c40b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1ad9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c1c071260 0x7f7c1c0716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1ad9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c1c071260 0x7f7c1c0716f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58454/0 (socket says 172.21.15.46:58454) 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1ad9d640 1 -- 172.21.15.46:0/1322477823 learned_addr learned my addr 172.21.15.46:0/1322477823 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1b59e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c1c15a7f0 con 0x7f7c1c071260 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1b59e640 1 -- 172.21.15.46:0/1322477823 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c071c30 con 0x7f7c1c156dd0 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f7c1a59c640 1 --2- 172.21.15.46:0/1322477823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c156dd0 0x7f7c1c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf7a477640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcf740b34a0 0x7fcf7415d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53262/0 (socket says 172.21.15.46:53262) 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf7a477640 1 -- 172.21.15.46:0/1253688823 learned_addr learned my addr 172.21.15.46:0/1253688823 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf78c74640 1 -- 172.21.15.46:0/1253688823 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081827472 0 0) 0x7fcf7415dc80 con 0x7fcf740b34a0 2023-12-16T21:13:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423801938 0 0) 0x7f178c0cd300 con 0x7f178c0b40b0 2023-12-16T21:13:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1778003520 con 0x7f178c0b40b0 2023-12-16T21:13:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f2dae59c640 1 --2- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da80e3090 0x7f2da8057d60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2da8056930 tx=0x7f2da402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b42170892e65c58 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7fcf78c74640 1 -- 172.21.15.46:0/1253688823 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf60003520 con 0x7fcf740b34a0 2023-12-16T21:13:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7fcf79c76640 1 --2- 172.21.15.46:0/1253688823 >> v2:172.21.15.46:3300/0 conn(0x7fcf74159210 0x7fcf7415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c2d9d640 1 -- 172.21.15.46:0/4165436310 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c40d1c30 con 0x7f92c409aef0 2023-12-16T21:13:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.564+0000 7f92c259c640 1 --2- 172.21.15.46:0/4165436310 >> v2:172.21.15.46:3300/0 conn(0x7f92c4164e80 0x7f92c406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476483393 0 0) 0x7f1778003520 con 0x7f178c0b40b0 2023-12-16T21:13:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 >> v2:172.21.15.46:3300/0 conn(0x7f178c0b1880 msgr2=0x7f178c0b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 --2- 172.21.15.46:0/111562721 >> v2:172.21.15.46:3300/0 conn(0x7f178c0b1880 0x7f178c0b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 >> v1:172.21.15.46:6789/0 conn(0x7f178c159210 legacy=0x7f178c15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1fb2d9d640 1 --2- 172.21.15.46:0/445448365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb4071640 0x7f1fb40cd3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009d90 tx=0x7f1fac02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c049157d7b2282db server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0cd5e0 con 0x7f178c0b40b0 2023-12-16T21:13:47.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f177c0015a0 con 0x7f178c0b40b0 2023-12-16T21:13:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f2dad59a640 1 -- 172.21.15.46:0/425470915 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402e070 con 0x7f2da80e3090 2023-12-16T21:13:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f2dad59a640 1 -- 172.21.15.46:0/425470915 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da40315a0 con 0x7f2da80e3090 2023-12-16T21:13:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f2dad59a640 1 -- 172.21.15.46:0/425470915 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402e070 con 0x7f2da80e3090 2023-12-16T21:13:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7fcf79c76640 1 -- 172.21.15.46:0/1253688823 >> v1:172.21.15.121:6789/0 conn(0x7fcf740b34a0 legacy=0x7fcf7415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7fcf79c76640 1 -- 172.21.15.46:0/1253688823 >> v1:172.21.15.46:6789/0 conn(0x7fcf740b40b0 legacy=0x7fcf7415ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7fcf79c76640 1 -- 172.21.15.46:0/1253688823 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf740cd300 con 0x7fcf74159210 2023-12-16T21:13:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 >> v1:172.21.15.121:6789/0 conn(0x7f178c0b40b0 legacy=0x7f178c0b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17780039c0 0x7f1778003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f92c2d9d640 1 --2- 172.21.15.46:0/4165436310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 0x7f92c409b340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009e50 tx=0x7f92bc02bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3510c00ae9574f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1778004320 0x7f17780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1778004f30 0x7f177800ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1fb159a640 1 -- 172.21.15.46:0/445448365 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac02cc60 con 0x7f1fb4071640 2023-12-16T21:13:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f92c159a640 1 -- 172.21.15.46:0/4165436310 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc02c960 con 0x7f92c409aef0 2023-12-16T21:13:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f92c159a640 1 -- 172.21.15.46:0/4165436310 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc02cac0 con 0x7f92c409aef0 2023-12-16T21:13:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f92c159a640 1 -- 172.21.15.46:0/4165436310 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc02ce50 con 0x7f92c409aef0 2023-12-16T21:13:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f2db4b41640 1 -- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da80e3090 msgr2=0x7f2da8057d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f2db4b41640 1 --2- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da80e3090 0x7f2da8057d60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2da8056930 tx=0x7f2da402b840 comp rx=0 tx=0).stop 2023-12-16T21:13:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0b4a70 con 0x7f1778004f30 2023-12-16T21:13:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17780104d0 con 0x7f1778004320 2023-12-16T21:13:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1778010640 con 0x7f17780039c0 2023-12-16T21:13:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.565+0000 7f1792049640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17780039c0 0x7f1778003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7fcf79c76640 1 --2- 172.21.15.46:0/1253688823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf74159210 0x7fcf7415a2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf700097a0 tx=0x7fcf7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8c534d4f338b7f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f179284a640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1778004f30 0x7f177800ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1792049640 1 -- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1778004320 msgr2=0x7f17780047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1792049640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1778004320 0x7f17780047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1792049640 1 -- 172.21.15.46:0/111562721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1778004f30 msgr2=0x7f177800ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1792049640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1778004f30 0x7f177800ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1792049640 1 -- 172.21.15.46:0/111562721 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0cd5e0 con 0x7f17780039c0 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f179284a640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1778004f30 0x7f177800ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f179284a640 1 -- 172.21.15.46:0/111562721 reap_dead start 2023-12-16T21:13:47.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f1fb159a640 1 -- 172.21.15.46:0/445448365 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fac02cdc0 con 0x7f1fb4071640 2023-12-16T21:13:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7fdc5fb43640 1 Processor -- start 2023-12-16T21:13:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7fdc5fb43640 1 -- start start 2023-12-16T21:13:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f7c1a59c640 1 -- 172.21.15.46:0/1322477823 >> v2:172.21.15.121:3300/0 conn(0x7f7c1c071260 msgr2=0x7f7c1c0716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f7c1a59c640 1 --2- 172.21.15.46:0/1322477823 >> v2:172.21.15.121:3300/0 conn(0x7f7c1c071260 0x7f7c1c0716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f7c1a59c640 1 -- 172.21.15.46:0/1322477823 >> v1:172.21.15.46:6789/0 conn(0x7f7c1c1597f0 legacy=0x7f7c1c0cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.566+0000 7f7c1a59c640 1 -- 172.21.15.46:0/1322477823 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c1571b0 con 0x7f7c1c156dd0 2023-12-16T21:13:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb159a640 1 -- 172.21.15.46:0/445448365 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fac02cc60 con 0x7f1fb4071640 2023-12-16T21:13:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f7c1a59c640 1 --2- 172.21.15.46:0/1322477823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c156dd0 0x7f7c1c0b3f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0097a0 tx=0x7f7c0c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63537c5beec67313 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f2db4b41640 1 -- 172.21.15.46:0/425470915 shutdown_connections 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f2db4b41640 1 --2- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da8054100 0x7f2da80e6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f2db4b41640 1 --2- 172.21.15.46:0/425470915 >> v2:172.21.15.46:3300/0 conn(0x7f2da803e810 0x7f2da8053bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f2db4b41640 1 --2- 172.21.15.46:0/425470915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da80e3090 0x7f2da8057d60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f2db4b41640 1 -- 172.21.15.46:0/425470915 >> 172.21.15.46:0/425470915 conn(0x7f2da80559b0 msgr2=0x7f2da8055dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1792049640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17780039c0 0x7f1778003de0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1788001810 tx=0x7f17780057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f31b4ba13f447a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7fcf78c74640 1 -- 172.21.15.46:0/1253688823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70031830 con 0x7fcf74159210 2023-12-16T21:13:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7fcf78c74640 1 -- 172.21.15.46:0/1253688823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70031990 con 0x7fcf74159210 2023-12-16T21:13:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f7c19d9b640 1 -- 172.21.15.46:0/1322477823 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c031830 con 0x7f7c1c156dd0 2023-12-16T21:13:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f7c19d9b640 1 -- 172.21.15.46:0/1322477823 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c031990 con 0x7f7c1c156dd0 2023-12-16T21:13:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788004aa0 con 0x7f17780039c0 2023-12-16T21:13:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f178802e070 con 0x7f17780039c0 2023-12-16T21:13:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1791047640 1 -- 172.21.15.46:0/111562721 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788004e40 con 0x7f17780039c0 2023-12-16T21:13:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f7c19d9b640 1 -- 172.21.15.46:0/1322477823 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c031d40 con 0x7f7c1c156dd0 2023-12-16T21:13:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb8a56640 1 -- 172.21.15.46:0/445448365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb4071640 msgr2=0x7f1fb40cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/445448365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb4071640 0x7f1fb40cd3f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009d90 tx=0x7f1fac02beb0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb8a56640 1 -- 172.21.15.46:0/445448365 shutdown_connections 2023-12-16T21:13:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/445448365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb4071640 0x7f1fb40cd3f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/445448365 >> v2:172.21.15.46:3300/0 conn(0x7f1fb414b050 0x7f1fb40b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f1fb8a56640 1 -- 172.21.15.46:0/445448365 >> 172.21.15.46:0/445448365 conn(0x7f1fb40b0d60 msgr2=0x7f1fb40b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f2db4b41640 1 -- 172.21.15.46:0/425470915 shutdown_connections 2023-12-16T21:13:47.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f92c89b0640 1 -- 172.21.15.46:0/4165436310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 msgr2=0x7f92c409b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f92c89b0640 1 --2- 172.21.15.46:0/4165436310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 0x7f92c409b340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009e50 tx=0x7f92bc02bdc0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f92c89b0640 1 -- 172.21.15.46:0/4165436310 shutdown_connections 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f92c89b0640 1 --2- 172.21.15.46:0/4165436310 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c409aef0 0x7f92c409b340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f92c89b0640 1 --2- 172.21.15.46:0/4165436310 >> v2:172.21.15.46:3300/0 conn(0x7f92c4164e80 0x7f92c406c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.567+0000 7f92c89b0640 1 -- 172.21.15.46:0/4165436310 >> 172.21.15.46:0/4165436310 conn(0x7f92c40d30d0 msgr2=0x7f92c40d34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f92c89b0640 1 -- 172.21.15.46:0/4165436310 shutdown_connections 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f7c1b59e640 1 -- 172.21.15.46:0/1322477823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c156dd0 msgr2=0x7f7c1c0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/1322477823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c156dd0 0x7f7c1c0b3f80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0097a0 tx=0x7f7c0c02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f1fb8a56640 1 -- 172.21.15.46:0/445448365 shutdown_connections 2023-12-16T21:13:47.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f2db4b41640 1 -- 172.21.15.46:0/425470915 wait complete. 2023-12-16T21:13:47.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17780039c0 msgr2=0x7f1778003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f17942ad640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17780039c0 0x7f1778003de0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1788001810 tx=0x7f17780057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5fb43640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdc581427d0 0x7fdc580b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5fb43640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdc580b4e80 0x7fdc5806da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5fb43640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc580cda40 con 0x7fdc58142270 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5fb43640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc580b5330 con 0x7fdc581427d0 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5fb43640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc580cbcc0 con 0x7fdc580b4e80 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fcf7beda640 1 -- 172.21.15.46:0/1253688823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf74159210 msgr2=0x7fcf7415a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fcf7beda640 1 --2- 172.21.15.46:0/1253688823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf74159210 0x7fcf7415a2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf700097a0 tx=0x7fcf7002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fcf7beda640 1 -- 172.21.15.46:0/1253688823 shutdown_connections 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fcf7beda640 1 --2- 172.21.15.46:0/1253688823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf74159210 0x7fcf7415a2f0 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcf700097a0 tx=0x7fcf7002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fcf7beda640 1 -- 172.21.15.46:0/1253688823 >> 172.21.15.46:0/1253688823 conn(0x7fcf740b0d60 msgr2=0x7fcf740b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5d8df640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fdc58142270 0x7fdc580cb5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:53270/0 (socket says 172.21.15.46:53270) 2023-12-16T21:13:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5d8df640 1 -- 172.21.15.46:0/1917916027 learned_addr learned my addr 172.21.15.46:0/1917916027 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 shutdown_connections 2023-12-16T21:13:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f17942ad640 1 --2- 172.21.15.46:0/111562721 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f17780039c0 0x7f1778003de0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 >> 172.21.15.46:0/111562721 conn(0x7f178c0b0d60 msgr2=0x7f178c0b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f1fb8a56640 1 -- 172.21.15.46:0/445448365 wait complete. 2023-12-16T21:13:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 shutdown_connections 2023-12-16T21:13:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7fdc5e0e0640 1 --2- 172.21.15.46:0/1917916027 >> v2:172.21.15.121:3300/0 conn(0x7fdc580b4e80 0x7fdc5806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f17942ad640 1 -- 172.21.15.46:0/111562721 wait complete. 2023-12-16T21:13:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680739103 0 0) 0x7fdc580cda40 con 0x7fdc58142270 2023-12-16T21:13:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc44003440 con 0x7fdc58142270 2023-12-16T21:13:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.568+0000 7f92c89b0640 1 -- 172.21.15.46:0/4165436310 wait complete. 2023-12-16T21:13:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fcf7beda640 1 -- 172.21.15.46:0/1253688823 shutdown_connections 2023-12-16T21:13:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3634052287 0 0) 0x7fdc44003440 con 0x7fdc58142270 2023-12-16T21:13:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 >> v2:172.21.15.121:3300/0 conn(0x7fdc580b4e80 msgr2=0x7fdc5806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 --2- 172.21.15.46:0/1917916027 >> v2:172.21.15.121:3300/0 conn(0x7fdc580b4e80 0x7fdc5806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 >> v2:172.21.15.46:3300/0 conn(0x7fdc581427d0 msgr2=0x7fdc580b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f7c1b59e640 1 -- 172.21.15.46:0/1322477823 shutdown_connections 2023-12-16T21:13:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/1322477823 >> v2:172.21.15.121:3300/0 conn(0x7f7c1c071260 0x7f7c1c0716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/1322477823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c156dd0 0x7f7c1c0b3f80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f7c1b59e640 1 -- 172.21.15.46:0/1322477823 >> 172.21.15.46:0/1322477823 conn(0x7f7c1c0b0c60 msgr2=0x7f7c1c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 --2- 172.21.15.46:0/1917916027 >> v2:172.21.15.46:3300/0 conn(0x7fdc581427d0 0x7fdc580b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f7c1b59e640 1 -- 172.21.15.46:0/1322477823 shutdown_connections 2023-12-16T21:13:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc580cdce0 con 0x7fdc58142270 2023-12-16T21:13:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fdc54004cd0 con 0x7fdc58142270 2023-12-16T21:13:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 >> v1:172.21.15.121:6789/0 conn(0x7fdc58142270 legacy=0x7fdc580cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f1fb8a56640 1 Processor -- start 2023-12-16T21:13:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f2db4b41640 1 Processor -- start 2023-12-16T21:13:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f92c89b0640 1 Processor -- start 2023-12-16T21:13:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f2db4b41640 1 -- start start 2023-12-16T21:13:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5c8dd640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc44003ab0 0x7fdc44003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5c8dd640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4400ac50 0x7fdc4400d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5c8dd640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc4400d760 0x7fdc4400fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f92c89b0640 1 -- start start 2023-12-16T21:13:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5d8df640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc44003ab0 0x7fdc44003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5e0e0640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc4400d760 0x7fdc4400fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc580b5330 con 0x7fdc4400ac50 2023-12-16T21:13:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5d0de640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4400ac50 0x7fdc4400d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc440104d0 con 0x7fdc4400d760 2023-12-16T21:13:47.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc44010690 con 0x7fdc44003ab0 2023-12-16T21:13:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7fcf7beda640 1 -- 172.21.15.46:0/1253688823 wait complete. 2023-12-16T21:13:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fcf7beda640 1 Processor -- start 2023-12-16T21:13:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fcf7beda640 1 -- start start 2023-12-16T21:13:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f1fb8a56640 1 -- start start 2023-12-16T21:13:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5d8df640 1 -- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc4400d760 msgr2=0x7fdc4400fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5d8df640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc4400d760 0x7fdc4400fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5d8df640 1 -- 172.21.15.46:0/1917916027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4400ac50 msgr2=0x7fdc4400d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f1fb8a56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb4071640 0x7f1fb406da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.569+0000 7f7c1b59e640 1 -- 172.21.15.46:0/1322477823 wait complete. 2023-12-16T21:13:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c89b0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 0x7f92c415eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c89b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c40b5f20 0x7f92c40c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f7c1b59e640 1 Processor -- start 2023-12-16T21:13:47.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f7c1b59e640 1 -- start start 2023-12-16T21:13:47.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1fb259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb4071640 0x7f1fb406da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1fb259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb4071640 0x7f1fb406da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36438/0 (socket says 172.21.15.46:36438) 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1fb259c640 1 -- 172.21.15.46:0/3915815814 learned_addr learned my addr 172.21.15.46:0/3915815814 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb40b4440 0x7f1fb4160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2db4b41640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da803e810 0x7f2da803d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2db4b41640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da8054100 0x7f2da803c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2db4b41640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da80e3090 0x7f2da803cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2db4b41640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80e56e0 con 0x7f2da803e810 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2db4b41640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80e4ad0 con 0x7f2da8054100 2023-12-16T21:13:47.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2db4b41640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2da80e4db0 con 0x7f2da80e3090 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f2daed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da80e3090 0x7f2da803cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f2dae59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da803e810 0x7f2da803d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f2daed9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da80e3090 0x7f2da803cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51334/0 (socket says 172.21.15.46:51334) 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f2daed9d640 1 -- 172.21.15.46:0/2271147091 learned_addr learned my addr 172.21.15.46:0/2271147091 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f17942ad640 1 Processor -- start 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.570+0000 7f17942ad640 1 -- start start 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f17942ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 0x7f178c1ee500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f17942ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f178c1474d0 0x7f178c15d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f17942ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f178c15d6d0 0x7f178c071770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f17942ad640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c148490 con 0x7f178c1474d0 2023-12-16T21:13:47.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1792049640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 0x7f178c1ee500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1792049640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 0x7f178c1ee500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58520/0 (socket says 172.21.15.46:58520) 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1792049640 1 -- 172.21.15.46:0/2603883337 learned_addr learned my addr 172.21.15.46:0/2603883337 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f179284a640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f178c15d6d0 0x7f178c071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c148770 con 0x7f17780039c0 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c148a50 con 0x7f178c15d6d0 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f1791848640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f178c1474d0 0x7f178c15d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5d8df640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4400ac50 0x7fdc4400d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7fdc5d8df640 1 -- 172.21.15.46:0/1917916027 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc44003440 con 0x7fdc44003ab0 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fdc5e0e0640 1 -- 172.21.15.46:0/1917916027 reap_dead start 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fdc5d0de640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc4400ac50 0x7fdc4400d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:47.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fdc5d8df640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc44003ab0 0x7fdc44003ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001e10 tx=0x7fdc440036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6341651b27a5b8dc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7fcf7beda640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf740b34a0 0x7fcf7413d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7fcf7beda640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf740b40b0 0x7fcf740b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7fcf7beda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 0x7fcf741ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf7beda640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf741ff8d0 con 0x7fcf7414ee70 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf7beda640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7415aa90 con 0x7fcf740b34a0 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf7beda640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7415ad30 con 0x7fcf740b40b0 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf7a477640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 0x7fcf741ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf7a477640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 0x7fcf741ff390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36458/0 (socket says 172.21.15.46:36458) 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf7a477640 1 -- 172.21.15.46:0/442214742 learned_addr learned my addr 172.21.15.46:0/442214742 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 0x7f92c415eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 0x7f92c415eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58508/0 (socket says 172.21.15.46:58508) 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c89b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c4164e80 0x7f92c40cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c1d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c40b5f20 0x7f92c40c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.571+0000 7f92c1d9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c40b5f20 0x7f92c40c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51316/0 (socket says 172.21.15.46:51316) 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 -- 172.21.15.46:0/1742045293 learned_addr learned my addr 172.21.15.46:0/1742045293 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c415f080 con 0x7f92c4164e80 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40c9ef0 con 0x7f92c409aef0 2023-12-16T21:13:47.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92c40c6810 con 0x7f92c40b5f20 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c2d9d640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c4164e80 0x7f92c40cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 -- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c40b5f20 msgr2=0x7f92c40c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c40b5f20 0x7f92c40c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 -- 172.21.15.46:0/1742045293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c4164e80 msgr2=0x7f92c40cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c4164e80 0x7f92c40cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc009b30 con 0x7f92c409aef0 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.572+0000 7f92c259c640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 0x7f92c415eb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f92b400c780 tx=0x7f92b400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4007a30 con 0x7f92c409aef0 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fb414b050 0x7f1fb4160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb41f6e80 con 0x7f1fb4071640 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb41fa510 con 0x7f1fb414b050 2023-12-16T21:13:47.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fb4161180 con 0x7f1fb40b4440 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1fb1d9b640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb40b4440 0x7f1fb4160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f2dadd9b640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da8054100 0x7f2da803c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1792049640 1 -- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f178c15d6d0 msgr2=0x7f178c071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1792049640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f178c15d6d0 0x7f178c071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1792049640 1 -- 172.21.15.46:0/2603883337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f178c1474d0 msgr2=0x7f178c15d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1792049640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f178c1474d0 0x7f178c15d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1792049640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c148d30 con 0x7f17780039c0 2023-12-16T21:13:47.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1791848640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f178c1474d0 0x7f178c15d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fcf79475640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf740b40b0 0x7fcf740b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1792049640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 0x7f178c1ee500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1788001930 tx=0x7f1788001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c40c6af0 con 0x7f92c409aef0 2023-12-16T21:13:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c40c70b0 con 0x7f92c409aef0 2023-12-16T21:13:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92b400ce90 con 0x7f92c409aef0 2023-12-16T21:13:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788007070 con 0x7f17780039c0 2023-12-16T21:13:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17880071d0 con 0x7f17780039c0 2023-12-16T21:13:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1788007540 con 0x7f17780039c0 2023-12-16T21:13:47.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f2dae59c640 1 -- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da80e3090 msgr2=0x7f2da803cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92b4007dd0 con 0x7f92c409aef0 2023-12-16T21:13:47.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f2dae59c640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da80e3090 0x7f2da803cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fcf7a477640 1 -- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf740b40b0 msgr2=0x7fcf740b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fcf7a477640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf740b40b0 0x7fcf740b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fcf7a477640 1 -- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf740b34a0 msgr2=0x7fcf7413d2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:13:47.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fcf7a477640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf740b34a0 0x7fcf7413d2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fcf7a477640 1 -- 172.21.15.46:0/442214742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf70009400 con 0x7fcf7414ee70 2023-12-16T21:13:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f2dae59c640 1 -- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da8054100 msgr2=0x7f2da803c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f2dae59c640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da8054100 0x7f2da803c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f7c1b59e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 0x7f7c1c0a8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f2dae59c640 1 -- 172.21.15.46:0/2271147091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da4009400 con 0x7f2da803e810 2023-12-16T21:13:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fcf7a477640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 0x7fcf741ff390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c001930 tx=0x7fcf6c001df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f7c1b59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1c156dd0 0x7f7c1c146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc54033620 con 0x7fdc44003ab0 2023-12-16T21:13:47.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.573+0000 7fdc5c8dd640 1 -- 172.21.15.46:0/1917916027 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5402f070 con 0x7fdc44003ab0 2023-12-16T21:13:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fdc5fb43640 1 -- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc44003ab0 msgr2=0x7fdc44003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc44003ab0 0x7fdc44003ed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001e10 tx=0x7fdc440036f0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 0x7f7c1c0a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f1fb2d9d640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fb414b050 0x7f1fb4160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f2dae59c640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da803e810 0x7f2da803d570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2da4033000 tx=0x7f2da40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f7c1a59c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 0x7f7c1c0a8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58542/0 (socket says 172.21.15.46:58542) 2023-12-16T21:13:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c149010 con 0x7f17780039c0 2023-12-16T21:13:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c147950 con 0x7f17780039c0 2023-12-16T21:13:47.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f7c1a59c640 1 -- 172.21.15.46:0/21623435 learned_addr learned my addr 172.21.15.46:0/21623435 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c1597f0 0x7f7c1c146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f1fb1d9b640 1 -- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fb414b050 msgr2=0x7f1fb4160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f1fb1d9b640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fb414b050 0x7f1fb4160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f1fb1d9b640 1 -- 172.21.15.46:0/3915815814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb4071640 msgr2=0x7f1fb406da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f1fb1d9b640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb4071640 0x7f1fb406da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.574+0000 7f1fb1d9b640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa80094d0 con 0x7f1fb40b4440 2023-12-16T21:13:47.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f1fb1d9b640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb40b4440 0x7f1fb4160700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c3c0 tx=0x7f1fa400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da402e070 con 0x7f2da803e810 2023-12-16T21:13:47.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c0082d0 con 0x7fcf7414ee70 2023-12-16T21:13:47.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c158be0 con 0x7f7c1c156dd0 2023-12-16T21:13:47.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7415b010 con 0x7fcf7414ee70 2023-12-16T21:13:47.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7415b5d0 con 0x7fcf7414ee70 2023-12-16T21:13:47.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6c00d6c0 con 0x7fcf7414ee70 2023-12-16T21:13:47.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f7c1ad9d640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c1597f0 0x7f7c1c146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2da80e5090 con 0x7f2da803e810 2023-12-16T21:13:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2da803dab0 con 0x7f2da803e810 2023-12-16T21:13:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c1471b0 con 0x7f7c1c071260 2023-12-16T21:13:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1c159000 con 0x7f7c1c1597f0 2023-12-16T21:13:47.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c00dab0 con 0x7fcf7414ee70 2023-12-16T21:13:47.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f92b40113f0 con 0x7f92c409aef0 2023-12-16T21:13:47.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f7c13fff640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1c156dd0 0x7f7c1c146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c069400 con 0x7f17780039c0 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c1a59c640 1 -- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c1597f0 msgr2=0x7f7c1c146c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c1a59c640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c1597f0 0x7f7c1c146c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c1a59c640 1 -- 172.21.15.46:0/21623435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1c156dd0 msgr2=0x7f7c1c146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c1a59c640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1c156dd0 0x7f7c1c146730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c1a59c640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0c009400 con 0x7f7c1c071260 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c13fff640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1c156dd0 0x7f7c1c146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:47.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.575+0000 7f2dadd9b640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da8054100 0x7f2da803c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da40325b0 con 0x7f2da803e810 2023-12-16T21:13:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da4032940 con 0x7f2da803e810 2023-12-16T21:13:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f7c1a59c640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 0x7f7c1c0a8740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0098c0 tx=0x7f7c0c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7fdc5fb43640 1 -- 172.21.15.46:0/1917916027 shutdown_connections 2023-12-16T21:13:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/1917916027 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc44003ab0 0x7fdc44003ed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7fdc5fb43640 1 -- 172.21.15.46:0/1917916027 >> 172.21.15.46:0/1917916027 conn(0x7fdc58070900 msgr2=0x7fdc58070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a8005250 con 0x7f92c409aef0 2023-12-16T21:13:47.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa40052d0 con 0x7f1fb40b4440 2023-12-16T21:13:47.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa4005450 con 0x7f1fb40b4440 2023-12-16T21:13:47.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7f92b2ffd640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92a406af10 0x7f92a406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fa4003830 con 0x7f1fb40b4440 2023-12-16T21:13:47.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5c005250 con 0x7fcf7414ee70 2023-12-16T21:13:47.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac009a70 con 0x7f1fb40b4440 2023-12-16T21:13:47.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fb41fa760 con 0x7f1fb40b4440 2023-12-16T21:13:47.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d94005250 con 0x7f2da803e810 2023-12-16T21:13:47.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.576+0000 7fdc5fb43640 1 -- 172.21.15.46:0/1917916027 shutdown_connections 2023-12-16T21:13:47.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7fdc5fb43640 1 -- 172.21.15.46:0/1917916027 wait complete. 2023-12-16T21:13:47.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7f92c1d9b640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92a406af10 0x7f92a406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.577+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92b4089630 con 0x7f92c409aef0 2023-12-16T21:13:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.578+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1fa4003990 con 0x7f1fb40b4440 2023-12-16T21:13:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.578+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb4069400 con 0x7f1fb40b4440 2023-12-16T21:13:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.578+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1788030070 con 0x7f17780039c0 2023-12-16T21:13:47.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f92c1d9b640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92a406af10 0x7f92a406d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92c40cc350 tx=0x7f92b801a070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2da4032aa0 con 0x7f2da803e810 2023-12-16T21:13:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7fdc5fb43640 1 Processor -- start 2023-12-16T21:13:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7fdc5fb43640 1 -- start start 2023-12-16T21:13:47.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f1fa2ffd640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f9406af10 0x7f1f9406d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fa4088ac0 con 0x7f1fb40b4440 2023-12-16T21:13:47.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f2d9effd640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d9806b050 0x7f2d9806d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fcf6c00dc10 con 0x7fcf7414ee70 2023-12-16T21:13:47.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2da40ab450 con 0x7f2da803e810 2023-12-16T21:13:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f1782ffd640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f177006ae90 0x7f177006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f17880ab600 con 0x7f17780039c0 2023-12-16T21:13:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f1791848640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f177006ae90 0x7f177006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.579+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c03b9f0 con 0x7f7c1c071260 2023-12-16T21:13:47.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7f1fb259c640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f9406af10 0x7f1f9406d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1c157f60 con 0x7f7c1c071260 2023-12-16T21:13:47.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fcf6a7fc640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcf5806ae90 0x7fcf5806d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1c158520 con 0x7f7c1c071260 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf6c0891f0 con 0x7fcf7414ee70 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7f2dadd9b640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d9806b050 0x7f2d9806d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5fb43640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc44003ab0 0x7fdc5813cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5fb43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc580a96a0 0x7fdc5815bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5fb43640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc5815c320 0x7fdc580aa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5fb43640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5815c7d0 con 0x7fdc580a96a0 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5fb43640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc580a8600 con 0x7fdc44003ab0 2023-12-16T21:13:47.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5fb43640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc580a88e0 con 0x7fdc5815c320 2023-12-16T21:13:47.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fcf79c76640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcf5806ae90 0x7fcf5806d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7f1fb259c640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f9406af10 0x7f1f9406d3d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009d80 tx=0x7f1fa8009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7f1791848640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f177006ae90 0x7f177006d350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f177c001110 tx=0x7f177c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5d8df640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc44003ab0 0x7fdc5813cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5e0e0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc5815c320 0x7fdc580aa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5d8df640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc44003ab0 0x7fdc5813cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:58550/0 (socket says 172.21.15.46:58550) 2023-12-16T21:13:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5d8df640 1 -- 172.21.15.46:0/3358541360 learned_addr learned my addr 172.21.15.46:0/3358541360 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:47.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.580+0000 7fdc5d0de640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc580a96a0 0x7fdc5815bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7f2dadd9b640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d9806b050 0x7f2d9806d510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2da8180690 tx=0x7f2da0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5e0e0640 1 -- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc44003ab0 msgr2=0x7fdc5813cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5e0e0640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc44003ab0 0x7fdc5813cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5e0e0640 1 -- 172.21.15.46:0/3358541360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc580a96a0 msgr2=0x7fdc5815bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fcf79c76640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcf5806ae90 0x7fcf5806d350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf70036000 tx=0x7fcf70008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5e0e0640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc580a96a0 0x7fdc5815bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5e0e0640 1 -- 172.21.15.46:0/3358541360 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc580a8bc0 con 0x7fdc5815c320 2023-12-16T21:13:47.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5d0de640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc580a96a0 0x7fdc5815bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1c158d20 con 0x7f7c1c071260 2023-12-16T21:13:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.581+0000 7fdc5e0e0640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc5815c320 0x7fdc580aa0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc5815cec0 tx=0x7fdc48011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.582+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f92b4054800 con 0x7f92c409aef0 2023-12-16T21:13:47.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.584+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48011ee0 con 0x7fdc5815c320 2023-12-16T21:13:47.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.584+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc48004240 con 0x7fdc5815c320 2023-12-16T21:13:47.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.584+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1fa4053c90 con 0x7f1fb40b4440 2023-12-16T21:13:47.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.584+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4800f770 con 0x7fdc5815c320 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.584+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc580a8ea0 con 0x7fdc5815c320 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.584+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc5813d080 con 0x7fdc5815c320 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.586+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc580a93c0 con 0x7fdc5815c320 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.586+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdc4800f8d0 con 0x7fdc5815c320 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.587+0000 7fdc4e7fc640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc3c06ae90 0x7fdc3c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.587+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c03bb50 con 0x7f7c1c071260 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.588+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c04d440 con 0x7f7c1c071260 2023-12-16T21:13:47.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.587+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1788076720 con 0x7f17780039c0 2023-12-16T21:13:47.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.587+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcf6c054430 con 0x7fcf7414ee70 2023-12-16T21:13:47.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.589+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7c0c04c3b0 con 0x7f7c1c071260 2023-12-16T21:13:47.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.587+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc48089520 con 0x7fdc5815c320 2023-12-16T21:13:47.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.590+0000 7fdc5d8df640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc3c06ae90 0x7fdc3c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.590+0000 7f7c12ffd640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7bfc06ae40 0x7f7bfc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.590+0000 7fdc5d8df640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc3c06ae90 0x7fdc3c06d350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001e10 tx=0x7fdc54030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.590+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c0c0ab4c0 con 0x7f7c1c071260 2023-12-16T21:13:47.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.591+0000 7f7c13fff640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7bfc06ae40 0x7f7bfc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:47.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.591+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdc480546f0 con 0x7fdc5815c320 2023-12-16T21:13:47.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.591+0000 7f7c13fff640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7bfc06ae40 0x7f7bfc06d300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c06e9f0 tx=0x7f7c04009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:47.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.592+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7c0c0767b0 con 0x7f7c1c071260 2023-12-16T21:13:47.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.596+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2da40764e0 con 0x7f2da803e810 2023-12-16T21:13:47.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.930+0000 7f2d3ec02640 1 -- 172.21.15.46:0/2746342677 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2d200057d0 con 0x7f2d300e78f0 2023-12-16T21:13:47.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.930+0000 7f2d2e7fc640 1 -- 172.21.15.46:0/2746342677 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2d380d1d10 con 0x7f2d300e78f0 2023-12-16T21:13:47.930 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.934+0000 7f2d1bfff640 1 -- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2406b0b0 msgr2=0x7f2d2406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.934+0000 7f2d1bfff640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2406b0b0 0x7f2d2406d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2d28034000 tx=0x7f2d28008030 comp rx=0 tx=0).stop 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.934+0000 7f2d1bfff640 1 -- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d300e78f0 msgr2=0x7f2d300cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.934+0000 7f2d1bfff640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d300e78f0 0x7f2d300cf4d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2d38068f20 tx=0x7f2d38067f00 comp rx=0 tx=0).stop 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 -- 172.21.15.46:0/2746342677 shutdown_connections 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2406b0b0 0x7f2d2406d570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d300e78f0 0x7f2d300cf4d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d300488d0 0x7f2d300e8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 --2- 172.21.15.46:0/2746342677 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d3003ac80 0x7f2d3004b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 -- 172.21.15.46:0/2746342677 >> 172.21.15.46:0/2746342677 conn(0x7f2d3001e690 msgr2=0x7f2d3000bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 -- 172.21.15.46:0/2746342677 shutdown_connections 2023-12-16T21:13:47.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.935+0000 7f2d1bfff640 1 -- 172.21.15.46:0/2746342677 wait complete. 2023-12-16T21:13:47.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2023-12-16T21:13:47.952 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:47.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.980+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1fb40b08f0 con 0x7f1fb40b4440 2023-12-16T21:13:47.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.981+0000 7f1fa2ffd640 1 -- 172.21.15.46:0/3915815814 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1fa4017090 con 0x7f1fb40b4440 2023-12-16T21:13:47.981 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:47.981 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.981+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2d94005500 con 0x7f2da803e810 2023-12-16T21:13:47.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.982+0000 7f2d9effd640 1 -- 172.21.15.46:0/2271147091 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2da4079b00 con 0x7f2da803e810 2023-12-16T21:13:47.982 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.983+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f9406af10 msgr2=0x7f1f9406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.983+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f9406af10 0x7f1f9406d3d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009d80 tx=0x7f1fa8009240 comp rx=0 tx=0).stop 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb40b4440 msgr2=0x7f1fb4160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb40b4440 0x7f1fb4160700 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fa400c3c0 tx=0x7f1fa400c880 comp rx=0 tx=0).stop 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 shutdown_connections 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f9406af10 0x7f1f9406d3d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fb414b050 0x7f1fb4160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1fb40b4440 0x7f1fb4160700 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 --2- 172.21.15.46:0/3915815814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1fb4071640 0x7f1fb406da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 >> 172.21.15.46:0/3915815814 conn(0x7f1fb40b0d60 msgr2=0x7f1fb40b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 shutdown_connections 2023-12-16T21:13:47.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.984+0000 7f1fb8a56640 1 -- 172.21.15.46:0/3915815814 wait complete. 2023-12-16T21:13:47.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d9806b050 msgr2=0x7f2d9806d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d9806b050 0x7f2d9806d510 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2da8180690 tx=0x7f2da0008040 comp rx=0 tx=0).stop 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da803e810 msgr2=0x7f2da803d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da803e810 0x7f2da803d570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2da4033000 tx=0x7f2da40046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 shutdown_connections 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d9806b050 0x7f2d9806d510 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2da80e3090 0x7f2da803cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2da8054100 0x7f2da803c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.985+0000 7f2db4b41640 1 --2- 172.21.15.46:0/2271147091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2da803e810 0x7f2da803d570 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.986+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 >> 172.21.15.46:0/2271147091 conn(0x7f2da80559b0 msgr2=0x7f2da8038990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.986+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 shutdown_connections 2023-12-16T21:13:47.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.986+0000 7f2db4b41640 1 -- 172.21.15.46:0/2271147091 wait complete. 2023-12-16T21:13:47.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.992+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f92a8005500 con 0x7f92c409aef0 2023-12-16T21:13:47.992 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:47.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.992+0000 7f92b2ffd640 1 -- 172.21.15.46:0/1742045293 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f92b4057e20 con 0x7f92c409aef0 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92a406af10 msgr2=0x7f92a406d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92a406af10 0x7f92a406d3d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f92c40cc350 tx=0x7f92b801a070 comp rx=0 tx=0).stop 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 msgr2=0x7f92c415eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 0x7f92c415eb40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f92b400c780 tx=0x7f92b400cc40 comp rx=0 tx=0).stop 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 shutdown_connections 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92a406af10 0x7f92a406d3d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92c4164e80 0x7f92c40cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92c40b5f20 0x7f92c40c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 --2- 172.21.15.46:0/1742045293 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92c409aef0 0x7f92c415eb40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:47.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.995+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 >> 172.21.15.46:0/1742045293 conn(0x7f92c40d30d0 msgr2=0x7f92c406c640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:47.997 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2023-12-16T21:13:47.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.997+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 shutdown_connections 2023-12-16T21:13:47.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:47.997+0000 7f92c89b0640 1 -- 172.21.15.46:0/1742045293 wait complete. 2023-12-16T21:13:47.997 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.000+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7c1c069320 con 0x7f7c1c071260 2023-12-16T21:13:48.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.001+0000 7f7c12ffd640 1 -- 172.21.15.46:0/21623435 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c0c079dd0 con 0x7f7c1c071260 2023-12-16T21:13:48.001 INFO:teuthology.orchestra.run.smithi046.stdout:55834574854 2023-12-16T21:13:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.001+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdc580a93c0 con 0x7fdc5815c320 2023-12-16T21:13:48.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.002+0000 7fdc4e7fc640 1 -- 172.21.15.46:0/3358541360 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc580a93c0 con 0x7fdc5815c320 2023-12-16T21:13:48.002 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:48.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2023-12-16T21:13:48.003 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7bfc06ae40 msgr2=0x7f7bfc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7bfc06ae40 0x7f7bfc06d300 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c06e9f0 tx=0x7f7c04009040 comp rx=0 tx=0).stop 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 msgr2=0x7f7c1c0a8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 0x7f7c1c0a8740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0098c0 tx=0x7f7c0c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 shutdown_connections 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7bfc06ae40 0x7f7bfc06d300 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c1c156dd0 0x7f7c1c146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c1c1597f0 0x7f7c1c146c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 --2- 172.21.15.46:0/21623435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c1c071260 0x7f7c1c0a8740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 >> 172.21.15.46:0/21623435 conn(0x7f7c1c0b0c60 msgr2=0x7f7c1c0aebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 shutdown_connections 2023-12-16T21:13:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f7c1b59e640 1 -- 172.21.15.46:0/21623435 wait complete. 2023-12-16T21:13:48.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.004+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f178c069400 con 0x7f17780039c0 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc3c06ae90 msgr2=0x7fdc3c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc3c06ae90 0x7fdc3c06d350 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc54001e10 tx=0x7fdc54030000 comp rx=0 tx=0).stop 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc5815c320 msgr2=0x7fdc580aa0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc5815c320 0x7fdc580aa0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdc5815cec0 tx=0x7fdc48011cf0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 shutdown_connections 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdc3c06ae90 0x7fdc3c06d350 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc580a96a0 0x7fdc5815bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc5815c320 0x7fdc580aa0f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 --2- 172.21.15.46:0/3358541360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc44003ab0 0x7fdc5813cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 >> 172.21.15.46:0/3358541360 conn(0x7fdc58070900 msgr2=0x7fdc5806e840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 shutdown_connections 2023-12-16T21:13:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7f1782ffd640 1 -- 172.21.15.46:0/2603883337 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1788079d40 con 0x7f17780039c0 2023-12-16T21:13:48.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fdc5fb43640 1 -- 172.21.15.46:0/3358541360 wait complete. 2023-12-16T21:13:48.008 INFO:teuthology.orchestra.run.smithi046.stdout:55834574856 2023-12-16T21:13:48.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcf5c0057d0 con 0x7fcf7414ee70 2023-12-16T21:13:48.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.005+0000 7fcf6a7fc640 1 -- 172.21.15.46:0/442214742 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcf6c022020 con 0x7fcf7414ee70 2023-12-16T21:13:48.008 INFO:teuthology.orchestra.run.smithi046.stdout:55834574855 2023-12-16T21:13:48.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f177006ae90 msgr2=0x7f177006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f177006ae90 0x7f177006d350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f177c001110 tx=0x7f177c009040 comp rx=0 tx=0).stop 2023-12-16T21:13:48.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 msgr2=0x7f178c1ee500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 0x7f178c1ee500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1788001930 tx=0x7f1788001120 comp rx=0 tx=0).stop 2023-12-16T21:13:48.009 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 shutdown_connections 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f177006ae90 0x7f177006d350 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f178c1474d0 0x7f178c15d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f178c15d6d0 0x7f178c071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 --2- 172.21.15.46:0/2603883337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17780039c0 0x7f178c1ee500 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1788001930 tx=0x7f1788001120 comp rx=0 tx=0).stop 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.007+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 >> 172.21.15.46:0/2603883337 conn(0x7f178c0b0d60 msgr2=0x7f178c0ce780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.008+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 shutdown_connections 2023-12-16T21:13:48.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.008+0000 7f17942ad640 1 -- 172.21.15.46:0/2603883337 wait complete. 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcf5806ae90 msgr2=0x7fcf5806d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcf5806ae90 0x7fcf5806d350 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf70036000 tx=0x7fcf70008030 comp rx=0 tx=0).stop 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 msgr2=0x7fcf741ff390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 0x7fcf741ff390 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c001930 tx=0x7fcf6c001df0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 shutdown_connections 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcf5806ae90 0x7fcf5806d350 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcf7414ee70 0x7fcf741ff390 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf740b40b0 0x7fcf740b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 --2- 172.21.15.46:0/442214742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf740b34a0 0x7fcf7413d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 >> 172.21.15.46:0/442214742 conn(0x7fcf740b0d60 msgr2=0x7fcf740ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 shutdown_connections 2023-12-16T21:13:48.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.009+0000 7fcf7beda640 1 -- 172.21.15.46:0/442214742 wait complete. 2023-12-16T21:13:48.012 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-12-16T21:13:48.012 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2023-12-16T21:13:48.020 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-12-16T21:13:48.021 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2023-12-16T21:13:48.022 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.023 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2023-12-16T21:13:48.023 DEBUG:teuthology.parallel:result is None 2023-12-16T21:13:48.023 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-16T21:13:48.023 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.170+0000 7f77f25a5640 1 Processor -- start 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.170+0000 7f77f25a5640 1 -- start start 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.170+0000 7f77f25a5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f77ec0d3900 0x7f77ec0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.170+0000 7f77f25a5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec0b2b10 con 0x7f77ec15d330 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.170+0000 7f77f25a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77ec0d1590 con 0x7f77ec0d2910 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.170+0000 7f77f25a5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77ec0d1870 con 0x7f77ec0d3900 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77ebfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f77ec15d330 0x7f77ec0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53564/0 (socket says 172.21.15.46:53564) 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77ebfff640 1 -- 172.21.15.46:0/1818447572 learned_addr learned my addr 172.21.15.46:0/1818447572 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77f0b42640 1 --2- 172.21.15.46:0/1818447572 >> v2:172.21.15.121:3300/0 conn(0x7f77ec0d3900 0x7f77ec0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247068607 0 0) 0x7f77ec0b2b10 con 0x7f77ec15d330 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77d4003520 con 0x7f77ec15d330 2023-12-16T21:13:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354471082 0 0) 0x7f77ec0d1590 con 0x7f77ec0d2910 2023-12-16T21:13:48.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77ec0b2b10 con 0x7f77ec0d2910 2023-12-16T21:13:48.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77f0b42640 1 -- 172.21.15.46:0/1818447572 >> v1:172.21.15.121:6789/0 conn(0x7f77ec0d2910 legacy=0x7f77ec0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77f0b42640 1 -- 172.21.15.46:0/1818447572 >> v1:172.21.15.46:6789/0 conn(0x7f77ec15d330 legacy=0x7f77ec0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.171+0000 7f77f0b42640 1 -- 172.21.15.46:0/1818447572 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec0d1b50 con 0x7f77ec0d3900 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.172+0000 7f77f0b42640 1 --2- 172.21.15.46:0/1818447572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec0d3900 0x7f77ec0b6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009d90 tx=0x7f77e402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16ab3e5fae6b80a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.172+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e402cc60 con 0x7f77ec0d3900 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.172+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e402cdc0 con 0x7f77ec0d3900 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.172+0000 7f77eaffd640 1 -- 172.21.15.46:0/1818447572 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4034640 con 0x7f77ec0d3900 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.172+0000 7f77f25a5640 1 -- 172.21.15.46:0/1818447572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec0d3900 msgr2=0x7f77ec0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.172+0000 7f77f25a5640 1 --2- 172.21.15.46:0/1818447572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec0d3900 0x7f77ec0b6390 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009d90 tx=0x7f77e402beb0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.173+0000 7f77f25a5640 1 -- 172.21.15.46:0/1818447572 shutdown_connections 2023-12-16T21:13:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.173+0000 7f77f25a5640 1 --2- 172.21.15.46:0/1818447572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec0d3900 0x7f77ec0b6390 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.173+0000 7f77f25a5640 1 -- 172.21.15.46:0/1818447572 >> 172.21.15.46:0/1818447572 conn(0x7f77ec072020 msgr2=0x7f77ec072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.173+0000 7f77f25a5640 1 -- 172.21.15.46:0/1818447572 shutdown_connections 2023-12-16T21:13:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.173+0000 7f77f25a5640 1 -- 172.21.15.46:0/1818447572 wait complete. 2023-12-16T21:13:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.174+0000 7f77f25a5640 1 Processor -- start 2023-12-16T21:13:48.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.174+0000 7f77f25a5640 1 -- start start 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77ec0d2910 0x7f77ec0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 0x7f77ec160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec15d330 0x7f77ec143380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0cbe30 con 0x7f77ec0d3900 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec1f2d70 con 0x7f77ec15d330 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec0c6ed0 con 0x7f77ec0d2910 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 0x7f77ec160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 0x7f77ec160520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36486/0 (socket says 172.21.15.46:36486) 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 -- 172.21.15.46:0/2677050172 learned_addr learned my addr 172.21.15.46:0/2677050172 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77ebfff640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77ec0d2910 0x7f77ec0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f0b42640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec15d330 0x7f77ec143380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 -- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77ec0d2910 msgr2=0x7f77ec0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77ec0d2910 0x7f77ec0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 -- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec15d330 msgr2=0x7f77ec143380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec15d330 0x7f77ec143380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 -- 172.21.15.46:0/2677050172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e4009a70 con 0x7f77ec0d3900 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77eb7fe640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 0x7f77ec160520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f77e000c500 tx=0x7f77e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e00052d0 con 0x7f77ec0d3900 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e0005450 con 0x7f77ec0d3900 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0012870 con 0x7f77ec0d3900 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.175+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77ec0cc0a0 con 0x7f77ec0d3900 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.176+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec0c6ba0 con 0x7f77ec0d3900 2023-12-16T21:13:48.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.177+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77ec069400 con 0x7f77ec0d3900 2023-12-16T21:13:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.177+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f77e00129d0 con 0x7f77ec0d3900 2023-12-16T21:13:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.177+0000 7f77e8ff9640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f77d806ae40 0x7f77d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.177+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77e0089240 con 0x7f77ec0d3900 2023-12-16T21:13:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.178+0000 7f77ebfff640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f77d806ae40 0x7f77d806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.178+0000 7f77ebfff640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f77d806ae40 0x7f77d806d300 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0011e0 tx=0x7f77dc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.182+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f77e0054300 con 0x7f77ec0d3900 2023-12-16T21:13:48.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.354+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77ec0c9ff0 con 0x7f77d806ae40 2023-12-16T21:13:48.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.361+0000 7f77e8ff9640 1 -- 172.21.15.46:0/2677050172 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264537 (secure 0 0 0) 0x7f77ec0c9ff0 con 0x7f77d806ae40 2023-12-16T21:13:48.363 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:48.365 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f77d806ae40 msgr2=0x7f77d806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f77d806ae40 0x7f77d806d300 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0011e0 tx=0x7f77dc009040 comp rx=0 tx=0).stop 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 msgr2=0x7f77ec160520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 0x7f77ec160520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f77e000c500 tx=0x7f77e000c9c0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 shutdown_connections 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f77d806ae40 0x7f77d806d300 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77ec0d3900 0x7f77ec160520 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77ec15d330 0x7f77ec143380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 --2- 172.21.15.46:0/2677050172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77ec0d2910 0x7f77ec0cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 >> 172.21.15.46:0/2677050172 conn(0x7f77ec072020 msgr2=0x7f77ec072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 shutdown_connections 2023-12-16T21:13:48.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.364+0000 7f77f25a5640 1 -- 172.21.15.46:0/2677050172 wait complete. 2023-12-16T21:13:48.374 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-12-16T21:13:47.957883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235752,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738968,"statfs":{"total":773094113280,"available":772852703232,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.474044"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580448+0000","last_change":"2023-12-16T21:13:36.497094+0000","last_active":"2023-12-16T21:13:37.580448+0000","last_peered":"2023-12-16T21:13:37.580448+0000","last_clean":"2023-12-16T21:13:37.580448+0000","last_became_active":"2023-12-16T21:13:36.496839+0000","last_became_peered":"2023-12-16T21:13:36.496839+0000","last_unstale":"2023-12-16T21:13:37.580448+0000","last_undegraded":"2023-12-16T21:13:37.580448+0000","last_fullsized":"2023-12-16T21:13:37.580448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043043+0000","last_change":"2023-12-16T21:13:36.496641+0000","last_active":"2023-12-16T21:13:38.043043+0000","last_peered":"2023-12-16T21:13:38.043043+0000","last_clean":"2023-12-16T21:13:38.043043+0000","last_became_active":"2023-12-16T21:13:36.496301+0000","last_became_peered":"2023-12-16T21:13:36.496301+0000","last_unstale":"2023-12-16T21:13:38.043043+0000","last_undegraded":"2023-12-16T21:13:38.043043+0000","last_fullsized":"2023-12-16T21:13:38.043043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004757+0000","last_change":"2023-12-16T21:13:36.490436+0000","last_active":"2023-12-16T21:13:38.004757+0000","last_peered":"2023-12-16T21:13:38.004757+0000","last_clean":"2023-12-16T21:13:38.004757+0000","last_became_active":"2023-12-16T21:13:36.489881+0000","last_became_peered":"2023-12-16T21:13:36.489881+0000","last_unstale":"2023-12-16T21:13:38.004757+0000","last_undegraded":"2023-12-16T21:13:38.004757+0000","last_fullsized":"2023-12-16T21:13:38.004757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487045+0000","last_change":"2023-12-16T21:13:36.507326+0000","last_active":"2023-12-16T21:13:37.487045+0000","last_peered":"2023-12-16T21:13:37.487045+0000","last_clean":"2023-12-16T21:13:37.487045+0000","last_became_active":"2023-12-16T21:13:36.507063+0000","last_became_peered":"2023-12-16T21:13:36.507063+0000","last_unstale":"2023-12-16T21:13:37.487045+0000","last_undegraded":"2023-12-16T21:13:37.487045+0000","last_fullsized":"2023-12-16T21:13:37.487045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580048+0000","last_change":"2023-12-16T21:13:36.494596+0000","last_active":"2023-12-16T21:13:37.580048+0000","last_peered":"2023-12-16T21:13:37.580048+0000","last_clean":"2023-12-16T21:13:37.580048+0000","last_became_active":"2023-12-16T21:13:36.494238+0000","last_became_peered":"2023-12-16T21:13:36.494238+0000","last_unstale":"2023-12-16T21:13:37.580048+0000","last_undegraded":"2023-12-16T21:13:37.580048+0000","last_fullsized":"2023-12-16T21:13:37.580048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486248+0000","last_change":"2023-12-16T21:13:36.505505+0000","last_active":"2023-12-16T21:13:37.486248+0000","last_peered":"2023-12-16T21:13:37.486248+0000","last_clean":"2023-12-16T21:13:37.486248+0000","last_became_active":"2023-12-16T21:13:36.505205+0000","last_became_peered":"2023-12-16T21:13:36.505205+0000","last_unstale":"2023-12-16T21:13:37.486248+0000","last_undegraded":"2023-12-16T21:13:37.486248+0000","last_fullsized":"2023-12-16T21:13:37.486248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579992+0000","last_change":"2023-12-16T21:13:36.502009+0000","last_active":"2023-12-16T21:13:37.579992+0000","last_peered":"2023-12-16T21:13:37.579992+0000","last_clean":"2023-12-16T21:13:37.579992+0000","last_became_active":"2023-12-16T21:13:36.501672+0000","last_became_peered":"2023-12-16T21:13:36.501672+0000","last_unstale":"2023-12-16T21:13:37.579992+0000","last_undegraded":"2023-12-16T21:13:37.579992+0000","last_fullsized":"2023-12-16T21:13:37.579992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.484971+0000","last_change":"2023-12-16T21:13:36.502551+0000","last_active":"2023-12-16T21:13:37.484971+0000","last_peered":"2023-12-16T21:13:37.484971+0000","last_clean":"2023-12-16T21:13:37.484971+0000","last_became_active":"2023-12-16T21:13:36.502214+0000","last_became_peered":"2023-12-16T21:13:36.502214+0000","last_unstale":"2023-12-16T21:13:37.484971+0000","last_undegraded":"2023-12-16T21:13:37.484971+0000","last_fullsized":"2023-12-16T21:13:37.484971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580474+0000","last_change":"2023-12-16T21:13:36.507546+0000","last_active":"2023-12-16T21:13:37.580474+0000","last_peered":"2023-12-16T21:13:37.580474+0000","last_clean":"2023-12-16T21:13:37.580474+0000","last_became_active":"2023-12-16T21:13:36.507253+0000","last_became_peered":"2023-12-16T21:13:36.507253+0000","last_unstale":"2023-12-16T21:13:37.580474+0000","last_undegraded":"2023-12-16T21:13:37.580474+0000","last_fullsized":"2023-12-16T21:13:37.580474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:22:24.104289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485901+0000","last_change":"2023-12-16T21:13:36.508602+0000","last_active":"2023-12-16T21:13:37.485901+0000","last_peered":"2023-12-16T21:13:37.485901+0000","last_clean":"2023-12-16T21:13:37.485901+0000","last_became_active":"2023-12-16T21:13:36.508223+0000","last_became_peered":"2023-12-16T21:13:36.508223+0000","last_unstale":"2023-12-16T21:13:37.485901+0000","last_undegraded":"2023-12-16T21:13:37.485901+0000","last_fullsized":"2023-12-16T21:13:37.485901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485806+0000","last_change":"2023-12-16T21:13:36.508624+0000","last_active":"2023-12-16T21:13:37.485806+0000","last_peered":"2023-12-16T21:13:37.485806+0000","last_clean":"2023-12-16T21:13:37.485806+0000","last_became_active":"2023-12-16T21:13:36.508293+0000","last_became_peered":"2023-12-16T21:13:36.508293+0000","last_unstale":"2023-12-16T21:13:37.485806+0000","last_undegraded":"2023-12-16T21:13:37.485806+0000","last_fullsized":"2023-12-16T21:13:37.485806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043125+0000","last_change":"2023-12-16T21:13:36.499195+0000","last_active":"2023-12-16T21:13:38.043125+0000","last_peered":"2023-12-16T21:13:38.043125+0000","last_clean":"2023-12-16T21:13:38.043125+0000","last_became_active":"2023-12-16T21:13:36.498927+0000","last_became_peered":"2023-12-16T21:13:36.498927+0000","last_unstale":"2023-12-16T21:13:38.043125+0000","last_undegraded":"2023-12-16T21:13:38.043125+0000","last_fullsized":"2023-12-16T21:13:38.043125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485287+0000","last_change":"2023-12-16T21:13:36.494278+0000","last_active":"2023-12-16T21:13:37.485287+0000","last_peered":"2023-12-16T21:13:37.485287+0000","last_clean":"2023-12-16T21:13:37.485287+0000","last_became_active":"2023-12-16T21:13:36.493970+0000","last_became_peered":"2023-12-16T21:13:36.493970+0000","last_unstale":"2023-12-16T21:13:37.485287+0000","last_undegraded":"2023-12-16T21:13:37.485287+0000","last_fullsized":"2023-12-16T21:13:37.485287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004516+0000","last_change":"2023-12-16T21:13:36.499374+0000","last_active":"2023-12-16T21:13:38.004516+0000","last_peered":"2023-12-16T21:13:38.004516+0000","last_clean":"2023-12-16T21:13:38.004516+0000","last_became_active":"2023-12-16T21:13:36.499097+0000","last_became_peered":"2023-12-16T21:13:36.499097+0000","last_unstale":"2023-12-16T21:13:38.004516+0000","last_undegraded":"2023-12-16T21:13:38.004516+0000","last_fullsized":"2023-12-16T21:13:38.004516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:33.841187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042953+0000","last_change":"2023-12-16T21:13:36.504350+0000","last_active":"2023-12-16T21:13:38.042953+0000","last_peered":"2023-12-16T21:13:38.042953+0000","last_clean":"2023-12-16T21:13:38.042953+0000","last_became_active":"2023-12-16T21:13:36.504046+0000","last_became_peered":"2023-12-16T21:13:36.504046+0000","last_unstale":"2023-12-16T21:13:38.042953+0000","last_undegraded":"2023-12-16T21:13:38.042953+0000","last_fullsized":"2023-12-16T21:13:38.042953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485909+0000","last_change":"2023-12-16T21:13:36.502440+0000","last_active":"2023-12-16T21:13:37.485909+0000","last_peered":"2023-12-16T21:13:37.485909+0000","last_clean":"2023-12-16T21:13:37.485909+0000","last_became_active":"2023-12-16T21:13:36.502160+0000","last_became_peered":"2023-12-16T21:13:36.502160+0000","last_unstale":"2023-12-16T21:13:37.485909+0000","last_undegraded":"2023-12-16T21:13:37.485909+0000","last_fullsized":"2023-12-16T21:13:37.485909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487000+0000","last_change":"2023-12-16T21:13:36.508778+0000","last_active":"2023-12-16T21:13:37.487000+0000","last_peered":"2023-12-16T21:13:37.487000+0000","last_clean":"2023-12-16T21:13:37.487000+0000","last_became_active":"2023-12-16T21:13:36.508506+0000","last_became_peered":"2023-12-16T21:13:36.508506+0000","last_unstale":"2023-12-16T21:13:37.487000+0000","last_undegraded":"2023-12-16T21:13:37.487000+0000","last_fullsized":"2023-12-16T21:13:37.487000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042744+0000","last_change":"2023-12-16T21:13:36.494459+0000","last_active":"2023-12-16T21:13:38.042744+0000","last_peered":"2023-12-16T21:13:38.042744+0000","last_clean":"2023-12-16T21:13:38.042744+0000","last_became_active":"2023-12-16T21:13:36.494156+0000","last_became_peered":"2023-12-16T21:13:36.494156+0000","last_unstale":"2023-12-16T21:13:38.042744+0000","last_undegraded":"2023-12-16T21:13:38.042744+0000","last_fullsized":"2023-12-16T21:13:38.042744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:47.886346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580395+0000","last_change":"2023-12-16T21:13:36.496446+0000","last_active":"2023-12-16T21:13:37.580395+0000","last_peered":"2023-12-16T21:13:37.580395+0000","last_clean":"2023-12-16T21:13:37.580395+0000","last_became_active":"2023-12-16T21:13:36.496123+0000","last_became_peered":"2023-12-16T21:13:36.496123+0000","last_unstale":"2023-12-16T21:13:37.580395+0000","last_undegraded":"2023-12-16T21:13:37.580395+0000","last_fullsized":"2023-12-16T21:13:37.580395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486172+0000","last_change":"2023-12-16T21:13:36.505223+0000","last_active":"2023-12-16T21:13:37.486172+0000","last_peered":"2023-12-16T21:13:37.486172+0000","last_clean":"2023-12-16T21:13:37.486172+0000","last_became_active":"2023-12-16T21:13:36.504834+0000","last_became_peered":"2023-12-16T21:13:36.504834+0000","last_unstale":"2023-12-16T21:13:37.486172+0000","last_undegraded":"2023-12-16T21:13:37.486172+0000","last_fullsized":"2023-12-16T21:13:37.486172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004625+0000","last_change":"2023-12-16T21:13:36.500880+0000","last_active":"2023-12-16T21:13:38.004625+0000","last_peered":"2023-12-16T21:13:38.004625+0000","last_clean":"2023-12-16T21:13:38.004625+0000","last_became_active":"2023-12-16T21:13:36.500532+0000","last_became_peered":"2023-12-16T21:13:36.500532+0000","last_unstale":"2023-12-16T21:13:38.004625+0000","last_undegraded":"2023-12-16T21:13:38.004625+0000","last_fullsized":"2023-12-16T21:13:38.004625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486491+0000","last_change":"2023-12-16T21:13:36.507578+0000","last_active":"2023-12-16T21:13:37.486491+0000","last_peered":"2023-12-16T21:13:37.486491+0000","last_clean":"2023-12-16T21:13:37.486491+0000","last_became_active":"2023-12-16T21:13:36.507206+0000","last_became_peered":"2023-12-16T21:13:36.507206+0000","last_unstale":"2023-12-16T21:13:37.486491+0000","last_undegraded":"2023-12-16T21:13:37.486491+0000","last_fullsized":"2023-12-16T21:13:37.486491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485167+0000","last_change":"2023-12-16T21:13:36.506029+0000","last_active":"2023-12-16T21:13:37.485167+0000","last_peered":"2023-12-16T21:13:37.485167+0000","last_clean":"2023-12-16T21:13:37.485167+0000","last_became_active":"2023-12-16T21:13:36.505785+0000","last_became_peered":"2023-12-16T21:13:36.505785+0000","last_unstale":"2023-12-16T21:13:37.485167+0000","last_undegraded":"2023-12-16T21:13:37.485167+0000","last_fullsized":"2023-12-16T21:13:37.485167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486054+0000","last_change":"2023-12-16T21:13:36.504916+0000","last_active":"2023-12-16T21:13:37.486054+0000","last_peered":"2023-12-16T21:13:37.486054+0000","last_clean":"2023-12-16T21:13:37.486054+0000","last_became_active":"2023-12-16T21:13:36.504422+0000","last_became_peered":"2023-12-16T21:13:36.504422+0000","last_unstale":"2023-12-16T21:13:37.486054+0000","last_undegraded":"2023-12-16T21:13:37.486054+0000","last_fullsized":"2023-12-16T21:13:37.486054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.503543+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.503210+0000","last_became_peered":"2023-12-16T21:13:36.503210+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486065+0000","last_change":"2023-12-16T21:13:36.494611+0000","last_active":"2023-12-16T21:13:37.486065+0000","last_peered":"2023-12-16T21:13:37.486065+0000","last_clean":"2023-12-16T21:13:37.486065+0000","last_became_active":"2023-12-16T21:13:36.494373+0000","last_became_peered":"2023-12-16T21:13:36.494373+0000","last_unstale":"2023-12-16T21:13:37.486065+0000","last_undegraded":"2023-12-16T21:13:37.486065+0000","last_fullsized":"2023-12-16T21:13:37.486065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043012+0000","last_change":"2023-12-16T21:13:36.498640+0000","last_active":"2023-12-16T21:13:38.043012+0000","last_peered":"2023-12-16T21:13:38.043012+0000","last_clean":"2023-12-16T21:13:38.043012+0000","last_became_active":"2023-12-16T21:13:36.498334+0000","last_became_peered":"2023-12-16T21:13:36.498334+0000","last_unstale":"2023-12-16T21:13:38.043012+0000","last_undegraded":"2023-12-16T21:13:38.043012+0000","last_fullsized":"2023-12-16T21:13:38.043012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485419+0000","last_change":"2023-12-16T21:13:36.506001+0000","last_active":"2023-12-16T21:13:37.485419+0000","last_peered":"2023-12-16T21:13:37.485419+0000","last_clean":"2023-12-16T21:13:37.485419+0000","last_became_active":"2023-12-16T21:13:36.505724+0000","last_became_peered":"2023-12-16T21:13:36.505724+0000","last_unstale":"2023-12-16T21:13:37.485419+0000","last_undegraded":"2023-12-16T21:13:37.485419+0000","last_fullsized":"2023-12-16T21:13:37.485419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004718+0000","last_change":"2023-12-16T21:13:36.489974+0000","last_active":"2023-12-16T21:13:38.004718+0000","last_peered":"2023-12-16T21:13:38.004718+0000","last_clean":"2023-12-16T21:13:38.004718+0000","last_became_active":"2023-12-16T21:13:36.489597+0000","last_became_peered":"2023-12-16T21:13:36.489597+0000","last_unstale":"2023-12-16T21:13:38.004718+0000","last_undegraded":"2023-12-16T21:13:38.004718+0000","last_fullsized":"2023-12-16T21:13:38.004718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486570+0000","last_change":"2023-12-16T21:13:36.505535+0000","last_active":"2023-12-16T21:13:37.486570+0000","last_peered":"2023-12-16T21:13:37.486570+0000","last_clean":"2023-12-16T21:13:37.486570+0000","last_became_active":"2023-12-16T21:13:36.505185+0000","last_became_peered":"2023-12-16T21:13:36.505185+0000","last_unstale":"2023-12-16T21:13:37.486570+0000","last_undegraded":"2023-12-16T21:13:37.486570+0000","last_fullsized":"2023-12-16T21:13:37.486570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042731+0000","last_change":"2023-12-16T21:13:36.504923+0000","last_active":"2023-12-16T21:13:38.042731+0000","last_peered":"2023-12-16T21:13:38.042731+0000","last_clean":"2023-12-16T21:13:38.042731+0000","last_became_active":"2023-12-16T21:13:36.504451+0000","last_became_peered":"2023-12-16T21:13:36.504451+0000","last_unstale":"2023-12-16T21:13:38.042731+0000","last_undegraded":"2023-12-16T21:13:38.042731+0000","last_fullsized":"2023-12-16T21:13:38.042731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485417+0000","last_change":"2023-12-16T21:13:36.497205+0000","last_active":"2023-12-16T21:13:37.485417+0000","last_peered":"2023-12-16T21:13:37.485417+0000","last_clean":"2023-12-16T21:13:37.485417+0000","last_became_active":"2023-12-16T21:13:36.496919+0000","last_became_peered":"2023-12-16T21:13:36.496919+0000","last_unstale":"2023-12-16T21:13:37.485417+0000","last_undegraded":"2023-12-16T21:13:37.485417+0000","last_fullsized":"2023-12-16T21:13:37.485417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042855+0000","last_change":"2023-12-16T21:13:36.503295+0000","last_active":"2023-12-16T21:13:38.042855+0000","last_peered":"2023-12-16T21:13:38.042855+0000","last_clean":"2023-12-16T21:13:38.042855+0000","last_became_active":"2023-12-16T21:13:36.502730+0000","last_became_peered":"2023-12-16T21:13:36.502730+0000","last_unstale":"2023-12-16T21:13:38.042855+0000","last_undegraded":"2023-12-16T21:13:38.042855+0000","last_fullsized":"2023-12-16T21:13:38.042855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485372+0000","last_change":"2023-12-16T21:13:35.508997+0000","last_active":"2023-12-16T21:13:37.485372+0000","last_peered":"2023-12-16T21:13:37.485372+0000","last_clean":"2023-12-16T21:13:37.485372+0000","last_became_active":"2023-12-16T21:13:35.508689+0000","last_became_peered":"2023-12-16T21:13:35.508689+0000","last_unstale":"2023-12-16T21:13:37.485372+0000","last_undegraded":"2023-12-16T21:13:37.485372+0000","last_fullsized":"2023-12-16T21:13:37.485372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004783+0000","last_change":"2023-12-16T21:13:36.504100+0000","last_active":"2023-12-16T21:13:38.004783+0000","last_peered":"2023-12-16T21:13:38.004783+0000","last_clean":"2023-12-16T21:13:38.004783+0000","last_became_active":"2023-12-16T21:13:36.503340+0000","last_became_peered":"2023-12-16T21:13:36.503340+0000","last_unstale":"2023-12-16T21:13:38.004783+0000","last_undegraded":"2023-12-16T21:13:38.004783+0000","last_fullsized":"2023-12-16T21:13:38.004783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485542+0000","last_change":"2023-12-16T21:13:35.508135+0000","last_active":"2023-12-16T21:13:37.485542+0000","last_peered":"2023-12-16T21:13:37.485542+0000","last_clean":"2023-12-16T21:13:37.485542+0000","last_became_active":"2023-12-16T21:13:35.507149+0000","last_became_peered":"2023-12-16T21:13:35.507149+0000","last_unstale":"2023-12-16T21:13:37.485542+0000","last_undegraded":"2023-12-16T21:13:37.485542+0000","last_fullsized":"2023-12-16T21:13:37.485542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485144+0000","last_change":"2023-12-16T21:13:35.510923+0000","last_active":"2023-12-16T21:13:37.485144+0000","last_peered":"2023-12-16T21:13:37.485144+0000","last_clean":"2023-12-16T21:13:37.485144+0000","last_became_active":"2023-12-16T21:13:35.510638+0000","last_became_peered":"2023-12-16T21:13:35.510638+0000","last_unstale":"2023-12-16T21:13:37.485144+0000","last_undegraded":"2023-12-16T21:13:37.485144+0000","last_fullsized":"2023-12-16T21:13:37.485144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:52.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485039+0000","last_change":"2023-12-16T21:13:36.490652+0000","last_active":"2023-12-16T21:13:37.485039+0000","last_peered":"2023-12-16T21:13:37.485039+0000","last_clean":"2023-12-16T21:13:37.485039+0000","last_became_active":"2023-12-16T21:13:36.489990+0000","last_became_peered":"2023-12-16T21:13:36.489990+0000","last_unstale":"2023-12-16T21:13:37.485039+0000","last_undegraded":"2023-12-16T21:13:37.485039+0000","last_fullsized":"2023-12-16T21:13:37.485039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486149+0000","last_change":"2023-12-16T21:13:35.498392+0000","last_active":"2023-12-16T21:13:37.486149+0000","last_peered":"2023-12-16T21:13:37.486149+0000","last_clean":"2023-12-16T21:13:37.486149+0000","last_became_active":"2023-12-16T21:13:35.498119+0000","last_became_peered":"2023-12-16T21:13:35.498119+0000","last_unstale":"2023-12-16T21:13:37.486149+0000","last_undegraded":"2023-12-16T21:13:37.486149+0000","last_fullsized":"2023-12-16T21:13:37.486149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042709+0000","last_change":"2023-12-16T21:13:36.499009+0000","last_active":"2023-12-16T21:13:38.042709+0000","last_peered":"2023-12-16T21:13:38.042709+0000","last_clean":"2023-12-16T21:13:38.042709+0000","last_became_active":"2023-12-16T21:13:36.498714+0000","last_became_peered":"2023-12-16T21:13:36.498714+0000","last_unstale":"2023-12-16T21:13:38.042709+0000","last_undegraded":"2023-12-16T21:13:38.042709+0000","last_fullsized":"2023-12-16T21:13:38.042709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485912+0000","last_change":"2023-12-16T21:13:35.508895+0000","last_active":"2023-12-16T21:13:37.485912+0000","last_peered":"2023-12-16T21:13:37.485912+0000","last_clean":"2023-12-16T21:13:37.485912+0000","last_became_active":"2023-12-16T21:13:35.508644+0000","last_became_peered":"2023-12-16T21:13:35.508644+0000","last_unstale":"2023-12-16T21:13:37.485912+0000","last_undegraded":"2023-12-16T21:13:37.485912+0000","last_fullsized":"2023-12-16T21:13:37.485912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486627+0000","last_change":"2023-12-16T21:13:36.508270+0000","last_active":"2023-12-16T21:13:37.486627+0000","last_peered":"2023-12-16T21:13:37.486627+0000","last_clean":"2023-12-16T21:13:37.486627+0000","last_became_active":"2023-12-16T21:13:36.507998+0000","last_became_peered":"2023-12-16T21:13:36.507998+0000","last_unstale":"2023-12-16T21:13:37.486627+0000","last_undegraded":"2023-12-16T21:13:37.486627+0000","last_fullsized":"2023-12-16T21:13:37.486627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486639+0000","last_change":"2023-12-16T21:13:35.508588+0000","last_active":"2023-12-16T21:13:37.486639+0000","last_peered":"2023-12-16T21:13:37.486639+0000","last_clean":"2023-12-16T21:13:37.486639+0000","last_became_active":"2023-12-16T21:13:35.508246+0000","last_became_peered":"2023-12-16T21:13:35.508246+0000","last_unstale":"2023-12-16T21:13:37.486639+0000","last_undegraded":"2023-12-16T21:13:37.486639+0000","last_fullsized":"2023-12-16T21:13:37.486639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004928+0000","last_change":"2023-12-16T21:13:36.507570+0000","last_active":"2023-12-16T21:13:38.004928+0000","last_peered":"2023-12-16T21:13:38.004928+0000","last_clean":"2023-12-16T21:13:38.004928+0000","last_became_active":"2023-12-16T21:13:36.507226+0000","last_became_peered":"2023-12-16T21:13:36.507226+0000","last_unstale":"2023-12-16T21:13:38.004928+0000","last_undegraded":"2023-12-16T21:13:38.004928+0000","last_fullsized":"2023-12-16T21:13:38.004928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968370+0000","last_change":"2023-12-16T21:13:35.509646+0000","last_active":"2023-12-16T21:13:37.968370+0000","last_peered":"2023-12-16T21:13:37.968370+0000","last_clean":"2023-12-16T21:13:37.968370+0000","last_became_active":"2023-12-16T21:13:35.509335+0000","last_became_peered":"2023-12-16T21:13:35.509335+0000","last_unstale":"2023-12-16T21:13:37.968370+0000","last_undegraded":"2023-12-16T21:13:37.968370+0000","last_fullsized":"2023-12-16T21:13:37.968370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043143+0000","last_change":"2023-12-16T21:13:36.499896+0000","last_active":"2023-12-16T21:13:38.043143+0000","last_peered":"2023-12-16T21:13:38.043143+0000","last_clean":"2023-12-16T21:13:38.043143+0000","last_became_active":"2023-12-16T21:13:36.499625+0000","last_became_peered":"2023-12-16T21:13:36.499625+0000","last_unstale":"2023-12-16T21:13:38.043143+0000","last_undegraded":"2023-12-16T21:13:38.043143+0000","last_fullsized":"2023-12-16T21:13:38.043143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486051+0000","last_change":"2023-12-16T21:13:35.507588+0000","last_active":"2023-12-16T21:13:37.486051+0000","last_peered":"2023-12-16T21:13:37.486051+0000","last_clean":"2023-12-16T21:13:37.486051+0000","last_became_active":"2023-12-16T21:13:35.507264+0000","last_became_peered":"2023-12-16T21:13:35.507264+0000","last_unstale":"2023-12-16T21:13:37.486051+0000","last_undegraded":"2023-12-16T21:13:37.486051+0000","last_fullsized":"2023-12-16T21:13:37.486051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580181+0000","last_change":"2023-12-16T21:13:36.495697+0000","last_active":"2023-12-16T21:13:37.580181+0000","last_peered":"2023-12-16T21:13:37.580181+0000","last_clean":"2023-12-16T21:13:37.580181+0000","last_became_active":"2023-12-16T21:13:36.495099+0000","last_became_peered":"2023-12-16T21:13:36.495099+0000","last_unstale":"2023-12-16T21:13:37.580181+0000","last_undegraded":"2023-12-16T21:13:37.580181+0000","last_fullsized":"2023-12-16T21:13:37.580181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968435+0000","last_change":"2023-12-16T21:13:35.502116+0000","last_active":"2023-12-16T21:13:37.968435+0000","last_peered":"2023-12-16T21:13:37.968435+0000","last_clean":"2023-12-16T21:13:37.968435+0000","last_became_active":"2023-12-16T21:13:35.501814+0000","last_became_peered":"2023-12-16T21:13:35.501814+0000","last_unstale":"2023-12-16T21:13:37.968435+0000","last_undegraded":"2023-12-16T21:13:37.968435+0000","last_fullsized":"2023-12-16T21:13:37.968435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004596+0000","last_change":"2023-12-16T21:13:36.499212+0000","last_active":"2023-12-16T21:13:38.004596+0000","last_peered":"2023-12-16T21:13:38.004596+0000","last_clean":"2023-12-16T21:13:38.004596+0000","last_became_active":"2023-12-16T21:13:36.498745+0000","last_became_peered":"2023-12-16T21:13:36.498745+0000","last_unstale":"2023-12-16T21:13:38.004596+0000","last_undegraded":"2023-12-16T21:13:38.004596+0000","last_fullsized":"2023-12-16T21:13:38.004596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:10.671619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486589+0000","last_change":"2023-12-16T21:13:35.506529+0000","last_active":"2023-12-16T21:13:37.486589+0000","last_peered":"2023-12-16T21:13:37.486589+0000","last_clean":"2023-12-16T21:13:37.486589+0000","last_became_active":"2023-12-16T21:13:35.506231+0000","last_became_peered":"2023-12-16T21:13:35.506231+0000","last_unstale":"2023-12-16T21:13:37.486589+0000","last_undegraded":"2023-12-16T21:13:37.486589+0000","last_fullsized":"2023-12-16T21:13:37.486589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580108+0000","last_change":"2023-12-16T21:13:36.504072+0000","last_active":"2023-12-16T21:13:37.580108+0000","last_peered":"2023-12-16T21:13:37.580108+0000","last_clean":"2023-12-16T21:13:37.580108+0000","last_became_active":"2023-12-16T21:13:36.503801+0000","last_became_peered":"2023-12-16T21:13:36.503801+0000","last_unstale":"2023-12-16T21:13:37.580108+0000","last_undegraded":"2023-12-16T21:13:37.580108+0000","last_fullsized":"2023-12-16T21:13:37.580108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485664+0000","last_change":"2023-12-16T21:13:35.508204+0000","last_active":"2023-12-16T21:13:37.485664+0000","last_peered":"2023-12-16T21:13:37.485664+0000","last_clean":"2023-12-16T21:13:37.485664+0000","last_became_active":"2023-12-16T21:13:35.507929+0000","last_became_peered":"2023-12-16T21:13:35.507929+0000","last_unstale":"2023-12-16T21:13:37.485664+0000","last_undegraded":"2023-12-16T21:13:37.485664+0000","last_fullsized":"2023-12-16T21:13:37.485664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485428+0000","last_change":"2023-12-16T21:13:36.497155+0000","last_active":"2023-12-16T21:13:37.485428+0000","last_peered":"2023-12-16T21:13:37.485428+0000","last_clean":"2023-12-16T21:13:37.485428+0000","last_became_active":"2023-12-16T21:13:36.496862+0000","last_became_peered":"2023-12-16T21:13:36.496862+0000","last_unstale":"2023-12-16T21:13:37.485428+0000","last_undegraded":"2023-12-16T21:13:37.485428+0000","last_fullsized":"2023-12-16T21:13:37.485428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968533+0000","last_change":"2023-12-16T21:13:35.508118+0000","last_active":"2023-12-16T21:13:37.968533+0000","last_peered":"2023-12-16T21:13:37.968533+0000","last_clean":"2023-12-16T21:13:37.968533+0000","last_became_active":"2023-12-16T21:13:35.507827+0000","last_became_peered":"2023-12-16T21:13:35.507827+0000","last_unstale":"2023-12-16T21:13:37.968533+0000","last_undegraded":"2023-12-16T21:13:37.968533+0000","last_fullsized":"2023-12-16T21:13:37.968533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580055+0000","last_change":"2023-12-16T21:13:36.504720+0000","last_active":"2023-12-16T21:13:37.580055+0000","last_peered":"2023-12-16T21:13:37.580055+0000","last_clean":"2023-12-16T21:13:37.580055+0000","last_became_active":"2023-12-16T21:13:36.504444+0000","last_became_peered":"2023-12-16T21:13:36.504444+0000","last_unstale":"2023-12-16T21:13:37.580055+0000","last_undegraded":"2023-12-16T21:13:37.580055+0000","last_fullsized":"2023-12-16T21:13:37.580055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968696+0000","last_change":"2023-12-16T21:13:35.509590+0000","last_active":"2023-12-16T21:13:37.968696+0000","last_peered":"2023-12-16T21:13:37.968696+0000","last_clean":"2023-12-16T21:13:37.968696+0000","last_became_active":"2023-12-16T21:13:35.509278+0000","last_became_peered":"2023-12-16T21:13:35.509278+0000","last_unstale":"2023-12-16T21:13:37.968696+0000","last_undegraded":"2023-12-16T21:13:37.968696+0000","last_fullsized":"2023-12-16T21:13:37.968696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:46.659974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580433+0000","last_change":"2023-12-16T21:13:36.498151+0000","last_active":"2023-12-16T21:13:37.580433+0000","last_peered":"2023-12-16T21:13:37.580433+0000","last_clean":"2023-12-16T21:13:37.580433+0000","last_became_active":"2023-12-16T21:13:36.497907+0000","last_became_peered":"2023-12-16T21:13:36.497907+0000","last_unstale":"2023-12-16T21:13:37.580433+0000","last_undegraded":"2023-12-16T21:13:37.580433+0000","last_fullsized":"2023-12-16T21:13:37.580433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968604+0000","last_change":"2023-12-16T21:13:35.507707+0000","last_active":"2023-12-16T21:13:37.968604+0000","last_peered":"2023-12-16T21:13:37.968604+0000","last_clean":"2023-12-16T21:13:37.968604+0000","last_became_active":"2023-12-16T21:13:35.507148+0000","last_became_peered":"2023-12-16T21:13:35.507148+0000","last_unstale":"2023-12-16T21:13:37.968604+0000","last_undegraded":"2023-12-16T21:13:37.968604+0000","last_fullsized":"2023-12-16T21:13:37.968604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580539+0000","last_change":"2023-12-16T21:13:36.503790+0000","last_active":"2023-12-16T21:13:37.580539+0000","last_peered":"2023-12-16T21:13:37.580539+0000","last_clean":"2023-12-16T21:13:37.580539+0000","last_became_active":"2023-12-16T21:13:36.503514+0000","last_became_peered":"2023-12-16T21:13:36.503514+0000","last_unstale":"2023-12-16T21:13:37.580539+0000","last_undegraded":"2023-12-16T21:13:37.580539+0000","last_fullsized":"2023-12-16T21:13:37.580539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:43:28.130579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968380+0000","last_change":"2023-12-16T21:13:35.507575+0000","last_active":"2023-12-16T21:13:37.968380+0000","last_peered":"2023-12-16T21:13:37.968380+0000","last_clean":"2023-12-16T21:13:37.968380+0000","last_became_active":"2023-12-16T21:13:35.507291+0000","last_became_peered":"2023-12-16T21:13:35.507291+0000","last_unstale":"2023-12-16T21:13:37.968380+0000","last_undegraded":"2023-12-16T21:13:37.968380+0000","last_fullsized":"2023-12-16T21:13:37.968380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487138+0000","last_change":"2023-12-16T21:13:36.507654+0000","last_active":"2023-12-16T21:13:37.487138+0000","last_peered":"2023-12-16T21:13:37.487138+0000","last_clean":"2023-12-16T21:13:37.487138+0000","last_became_active":"2023-12-16T21:13:36.507245+0000","last_became_peered":"2023-12-16T21:13:36.507245+0000","last_unstale":"2023-12-16T21:13:37.487138+0000","last_undegraded":"2023-12-16T21:13:37.487138+0000","last_fullsized":"2023-12-16T21:13:37.487138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968585+0000","last_change":"2023-12-16T21:13:35.509065+0000","last_active":"2023-12-16T21:13:37.968585+0000","last_peered":"2023-12-16T21:13:37.968585+0000","last_clean":"2023-12-16T21:13:37.968585+0000","last_became_active":"2023-12-16T21:13:35.508808+0000","last_became_peered":"2023-12-16T21:13:35.508808+0000","last_unstale":"2023-12-16T21:13:37.968585+0000","last_undegraded":"2023-12-16T21:13:37.968585+0000","last_fullsized":"2023-12-16T21:13:37.968585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580284+0000","last_change":"2023-12-16T21:13:36.507427+0000","last_active":"2023-12-16T21:13:37.580284+0000","last_peered":"2023-12-16T21:13:37.580284+0000","last_clean":"2023-12-16T21:13:37.580284+0000","last_became_active":"2023-12-16T21:13:36.507170+0000","last_became_peered":"2023-12-16T21:13:36.507170+0000","last_unstale":"2023-12-16T21:13:37.580284+0000","last_undegraded":"2023-12-16T21:13:37.580284+0000","last_fullsized":"2023-12-16T21:13:37.580284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490630+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:36.490630+0000","last_peered":"2023-12-16T21:13:36.490630+0000","last_clean":"2023-12-16T21:13:36.490630+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:36.490630+0000","last_undegraded":"2023-12-16T21:13:36.490630+0000","last_fullsized":"2023-12-16T21:13:36.490630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486371+0000","last_change":"2023-12-16T21:13:35.505945+0000","last_active":"2023-12-16T21:13:37.486371+0000","last_peered":"2023-12-16T21:13:37.486371+0000","last_clean":"2023-12-16T21:13:37.486371+0000","last_became_active":"2023-12-16T21:13:35.505670+0000","last_became_peered":"2023-12-16T21:13:35.505670+0000","last_unstale":"2023-12-16T21:13:37.486371+0000","last_undegraded":"2023-12-16T21:13:37.486371+0000","last_fullsized":"2023-12-16T21:13:37.486371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485341+0000","last_change":"2023-12-16T21:13:36.507596+0000","last_active":"2023-12-16T21:13:37.485341+0000","last_peered":"2023-12-16T21:13:37.485341+0000","last_clean":"2023-12-16T21:13:37.485341+0000","last_became_active":"2023-12-16T21:13:36.507302+0000","last_became_peered":"2023-12-16T21:13:36.507302+0000","last_unstale":"2023-12-16T21:13:37.485341+0000","last_undegraded":"2023-12-16T21:13:37.485341+0000","last_fullsized":"2023-12-16T21:13:37.485341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579838+0000","last_change":"2023-12-16T21:13:35.512856+0000","last_active":"2023-12-16T21:13:37.579838+0000","last_peered":"2023-12-16T21:13:37.579838+0000","last_clean":"2023-12-16T21:13:37.579838+0000","last_became_active":"2023-12-16T21:13:35.512539+0000","last_became_peered":"2023-12-16T21:13:35.512539+0000","last_unstale":"2023-12-16T21:13:37.579838+0000","last_undegraded":"2023-12-16T21:13:37.579838+0000","last_fullsized":"2023-12-16T21:13:37.579838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004529+0000","last_change":"2023-12-16T21:13:36.503724+0000","last_active":"2023-12-16T21:13:38.004529+0000","last_peered":"2023-12-16T21:13:38.004529+0000","last_clean":"2023-12-16T21:13:38.004529+0000","last_became_active":"2023-12-16T21:13:36.503392+0000","last_became_peered":"2023-12-16T21:13:36.503392+0000","last_unstale":"2023-12-16T21:13:38.004529+0000","last_undegraded":"2023-12-16T21:13:38.004529+0000","last_fullsized":"2023-12-16T21:13:38.004529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:02.640523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968573+0000","last_change":"2023-12-16T21:13:35.506777+0000","last_active":"2023-12-16T21:13:37.968573+0000","last_peered":"2023-12-16T21:13:37.968573+0000","last_clean":"2023-12-16T21:13:37.968573+0000","last_became_active":"2023-12-16T21:13:35.506483+0000","last_became_peered":"2023-12-16T21:13:35.506483+0000","last_unstale":"2023-12-16T21:13:37.968573+0000","last_undegraded":"2023-12-16T21:13:37.968573+0000","last_fullsized":"2023-12-16T21:13:37.968573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:25.556208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042790+0000","last_change":"2023-12-16T21:13:36.496004+0000","last_active":"2023-12-16T21:13:38.042790+0000","last_peered":"2023-12-16T21:13:38.042790+0000","last_clean":"2023-12-16T21:13:38.042790+0000","last_became_active":"2023-12-16T21:13:36.495712+0000","last_became_peered":"2023-12-16T21:13:36.495712+0000","last_unstale":"2023-12-16T21:13:38.042790+0000","last_undegraded":"2023-12-16T21:13:38.042790+0000","last_fullsized":"2023-12-16T21:13:38.042790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486820+0000","last_change":"2023-12-16T21:13:35.507557+0000","last_active":"2023-12-16T21:13:37.486820+0000","last_peered":"2023-12-16T21:13:37.486820+0000","last_clean":"2023-12-16T21:13:37.486820+0000","last_became_active":"2023-12-16T21:13:35.507175+0000","last_became_peered":"2023-12-16T21:13:35.507175+0000","last_unstale":"2023-12-16T21:13:37.486820+0000","last_undegraded":"2023-12-16T21:13:37.486820+0000","last_fullsized":"2023-12-16T21:13:37.486820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:11.664911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043017+0000","last_change":"2023-12-16T21:13:36.502502+0000","last_active":"2023-12-16T21:13:38.043017+0000","last_peered":"2023-12-16T21:13:38.043017+0000","last_clean":"2023-12-16T21:13:38.043017+0000","last_became_active":"2023-12-16T21:13:36.501893+0000","last_became_peered":"2023-12-16T21:13:36.501893+0000","last_unstale":"2023-12-16T21:13:38.043017+0000","last_undegraded":"2023-12-16T21:13:38.043017+0000","last_fullsized":"2023-12-16T21:13:38.043017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:46.012193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968352+0000","last_change":"2023-12-16T21:13:35.513556+0000","last_active":"2023-12-16T21:13:37.968352+0000","last_peered":"2023-12-16T21:13:37.968352+0000","last_clean":"2023-12-16T21:13:37.968352+0000","last_became_active":"2023-12-16T21:13:35.513208+0000","last_became_peered":"2023-12-16T21:13:35.513208+0000","last_unstale":"2023-12-16T21:13:37.968352+0000","last_undegraded":"2023-12-16T21:13:37.968352+0000","last_fullsized":"2023-12-16T21:13:37.968352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486956+0000","last_change":"2023-12-16T21:13:36.504553+0000","last_active":"2023-12-16T21:13:37.486956+0000","last_peered":"2023-12-16T21:13:37.486956+0000","last_clean":"2023-12-16T21:13:37.486956+0000","last_became_active":"2023-12-16T21:13:36.504276+0000","last_became_peered":"2023-12-16T21:13:36.504276+0000","last_unstale":"2023-12-16T21:13:37.486956+0000","last_undegraded":"2023-12-16T21:13:37.486956+0000","last_fullsized":"2023-12-16T21:13:37.486956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580388+0000","last_change":"2023-12-16T21:13:35.511628+0000","last_active":"2023-12-16T21:13:37.580388+0000","last_peered":"2023-12-16T21:13:37.580388+0000","last_clean":"2023-12-16T21:13:37.580388+0000","last_became_active":"2023-12-16T21:13:35.510934+0000","last_became_peered":"2023-12-16T21:13:35.510934+0000","last_unstale":"2023-12-16T21:13:37.580388+0000","last_undegraded":"2023-12-16T21:13:37.580388+0000","last_fullsized":"2023-12-16T21:13:37.580388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:21.664872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043123+0000","last_change":"2023-12-16T21:13:36.495330+0000","last_active":"2023-12-16T21:13:38.043123+0000","last_peered":"2023-12-16T21:13:38.043123+0000","last_clean":"2023-12-16T21:13:38.043123+0000","last_became_active":"2023-12-16T21:13:36.495064+0000","last_became_peered":"2023-12-16T21:13:36.495064+0000","last_unstale":"2023-12-16T21:13:38.043123+0000","last_undegraded":"2023-12-16T21:13:38.043123+0000","last_fullsized":"2023-12-16T21:13:38.043123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043073+0000","last_change":"2023-12-16T21:13:35.508842+0000","last_active":"2023-12-16T21:13:38.043073+0000","last_peered":"2023-12-16T21:13:38.043073+0000","last_clean":"2023-12-16T21:13:38.043073+0000","last_became_active":"2023-12-16T21:13:35.508552+0000","last_became_peered":"2023-12-16T21:13:35.508552+0000","last_unstale":"2023-12-16T21:13:38.043073+0000","last_undegraded":"2023-12-16T21:13:38.043073+0000","last_fullsized":"2023-12-16T21:13:38.043073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.497136+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.496859+0000","last_became_peered":"2023-12-16T21:13:36.496859+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486007+0000","last_change":"2023-12-16T21:13:35.512677+0000","last_active":"2023-12-16T21:13:37.486007+0000","last_peered":"2023-12-16T21:13:37.486007+0000","last_clean":"2023-12-16T21:13:37.486007+0000","last_became_active":"2023-12-16T21:13:35.511936+0000","last_became_peered":"2023-12-16T21:13:35.511936+0000","last_unstale":"2023-12-16T21:13:37.486007+0000","last_undegraded":"2023-12-16T21:13:37.486007+0000","last_fullsized":"2023-12-16T21:13:37.486007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004860+0000","last_change":"2023-12-16T21:13:36.499822+0000","last_active":"2023-12-16T21:13:38.004860+0000","last_peered":"2023-12-16T21:13:38.004860+0000","last_clean":"2023-12-16T21:13:38.004860+0000","last_became_active":"2023-12-16T21:13:36.499517+0000","last_became_peered":"2023-12-16T21:13:36.499517+0000","last_unstale":"2023-12-16T21:13:38.004860+0000","last_undegraded":"2023-12-16T21:13:38.004860+0000","last_fullsized":"2023-12-16T21:13:38.004860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579837+0000","last_change":"2023-12-16T21:13:35.511783+0000","last_active":"2023-12-16T21:13:37.579837+0000","last_peered":"2023-12-16T21:13:37.579837+0000","last_clean":"2023-12-16T21:13:37.579837+0000","last_became_active":"2023-12-16T21:13:35.510752+0000","last_became_peered":"2023-12-16T21:13:35.510752+0000","last_unstale":"2023-12-16T21:13:37.579837+0000","last_undegraded":"2023-12-16T21:13:37.579837+0000","last_fullsized":"2023-12-16T21:13:37.579837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043007+0000","last_change":"2023-12-16T21:13:36.503029+0000","last_active":"2023-12-16T21:13:38.043007+0000","last_peered":"2023-12-16T21:13:38.043007+0000","last_clean":"2023-12-16T21:13:38.043007+0000","last_became_active":"2023-12-16T21:13:36.502641+0000","last_became_peered":"2023-12-16T21:13:36.502641+0000","last_unstale":"2023-12-16T21:13:38.043007+0000","last_undegraded":"2023-12-16T21:13:38.043007+0000","last_fullsized":"2023-12-16T21:13:38.043007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968311+0000","last_change":"2023-12-16T21:13:35.511492+0000","last_active":"2023-12-16T21:13:37.968311+0000","last_peered":"2023-12-16T21:13:37.968311+0000","last_clean":"2023-12-16T21:13:37.968311+0000","last_became_active":"2023-12-16T21:13:35.511118+0000","last_became_peered":"2023-12-16T21:13:35.511118+0000","last_unstale":"2023-12-16T21:13:37.968311+0000","last_undegraded":"2023-12-16T21:13:37.968311+0000","last_fullsized":"2023-12-16T21:13:37.968311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485112+0000","last_change":"2023-12-16T21:13:36.495196+0000","last_active":"2023-12-16T21:13:37.485112+0000","last_peered":"2023-12-16T21:13:37.485112+0000","last_clean":"2023-12-16T21:13:37.485112+0000","last_became_active":"2023-12-16T21:13:36.494933+0000","last_became_peered":"2023-12-16T21:13:36.494933+0000","last_unstale":"2023-12-16T21:13:37.485112+0000","last_undegraded":"2023-12-16T21:13:37.485112+0000","last_fullsized":"2023-12-16T21:13:37.485112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968321+0000","last_change":"2023-12-16T21:13:35.506652+0000","last_active":"2023-12-16T21:13:37.968321+0000","last_peered":"2023-12-16T21:13:37.968321+0000","last_clean":"2023-12-16T21:13:37.968321+0000","last_became_active":"2023-12-16T21:13:35.506349+0000","last_became_peered":"2023-12-16T21:13:35.506349+0000","last_unstale":"2023-12-16T21:13:37.968321+0000","last_undegraded":"2023-12-16T21:13:37.968321+0000","last_fullsized":"2023-12-16T21:13:37.968321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485338+0000","last_change":"2023-12-16T21:13:36.490929+0000","last_active":"2023-12-16T21:13:37.485338+0000","last_peered":"2023-12-16T21:13:37.485338+0000","last_clean":"2023-12-16T21:13:37.485338+0000","last_became_active":"2023-12-16T21:13:36.490557+0000","last_became_peered":"2023-12-16T21:13:36.490557+0000","last_unstale":"2023-12-16T21:13:37.485338+0000","last_undegraded":"2023-12-16T21:13:37.485338+0000","last_fullsized":"2023-12-16T21:13:37.485338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580200+0000","last_change":"2023-12-16T21:13:35.511062+0000","last_active":"2023-12-16T21:13:37.580200+0000","last_peered":"2023-12-16T21:13:37.580200+0000","last_clean":"2023-12-16T21:13:37.580200+0000","last_became_active":"2023-12-16T21:13:35.510780+0000","last_became_peered":"2023-12-16T21:13:35.510780+0000","last_unstale":"2023-12-16T21:13:37.580200+0000","last_undegraded":"2023-12-16T21:13:37.580200+0000","last_fullsized":"2023-12-16T21:13:37.580200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:25.707555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485439+0000","last_change":"2023-12-16T21:13:36.499546+0000","last_active":"2023-12-16T21:13:37.485439+0000","last_peered":"2023-12-16T21:13:37.485439+0000","last_clean":"2023-12-16T21:13:37.485439+0000","last_became_active":"2023-12-16T21:13:36.499236+0000","last_became_peered":"2023-12-16T21:13:36.499236+0000","last_unstale":"2023-12-16T21:13:37.485439+0000","last_undegraded":"2023-12-16T21:13:37.485439+0000","last_fullsized":"2023-12-16T21:13:37.485439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486745+0000","last_change":"2023-12-16T21:13:36.512546+0000","last_active":"2023-12-16T21:13:37.486745+0000","last_peered":"2023-12-16T21:13:37.486745+0000","last_clean":"2023-12-16T21:13:37.486745+0000","last_became_active":"2023-12-16T21:13:36.512196+0000","last_became_peered":"2023-12-16T21:13:36.512196+0000","last_unstale":"2023-12-16T21:13:37.486745+0000","last_undegraded":"2023-12-16T21:13:37.486745+0000","last_fullsized":"2023-12-16T21:13:37.486745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580134+0000","last_change":"2023-12-16T21:13:35.511999+0000","last_active":"2023-12-16T21:13:37.580134+0000","last_peered":"2023-12-16T21:13:37.580134+0000","last_clean":"2023-12-16T21:13:37.580134+0000","last_became_active":"2023-12-16T21:13:35.511760+0000","last_became_peered":"2023-12-16T21:13:35.511760+0000","last_unstale":"2023-12-16T21:13:37.580134+0000","last_undegraded":"2023-12-16T21:13:37.580134+0000","last_fullsized":"2023-12-16T21:13:37.580134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:49.269497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487061+0000","last_change":"2023-12-16T21:13:36.512673+0000","last_active":"2023-12-16T21:13:37.487061+0000","last_peered":"2023-12-16T21:13:37.487061+0000","last_clean":"2023-12-16T21:13:37.487061+0000","last_became_active":"2023-12-16T21:13:36.512185+0000","last_became_peered":"2023-12-16T21:13:36.512185+0000","last_unstale":"2023-12-16T21:13:37.487061+0000","last_undegraded":"2023-12-16T21:13:37.487061+0000","last_fullsized":"2023-12-16T21:13:37.487061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580255+0000","last_change":"2023-12-16T21:13:35.511982+0000","last_active":"2023-12-16T21:13:37.580255+0000","last_peered":"2023-12-16T21:13:37.580255+0000","last_clean":"2023-12-16T21:13:37.580255+0000","last_became_active":"2023-12-16T21:13:35.511359+0000","last_became_peered":"2023-12-16T21:13:35.511359+0000","last_unstale":"2023-12-16T21:13:37.580255+0000","last_undegraded":"2023-12-16T21:13:37.580255+0000","last_fullsized":"2023-12-16T21:13:37.580255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004907+0000","last_change":"2023-12-16T21:13:36.505890+0000","last_active":"2023-12-16T21:13:38.004907+0000","last_peered":"2023-12-16T21:13:38.004907+0000","last_clean":"2023-12-16T21:13:38.004907+0000","last_became_active":"2023-12-16T21:13:36.505414+0000","last_became_peered":"2023-12-16T21:13:36.505414+0000","last_unstale":"2023-12-16T21:13:38.004907+0000","last_undegraded":"2023-12-16T21:13:38.004907+0000","last_fullsized":"2023-12-16T21:13:38.004907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485644+0000","last_change":"2023-12-16T21:13:35.511468+0000","last_active":"2023-12-16T21:13:37.485644+0000","last_peered":"2023-12-16T21:13:37.485644+0000","last_clean":"2023-12-16T21:13:37.485644+0000","last_became_active":"2023-12-16T21:13:35.511092+0000","last_became_peered":"2023-12-16T21:13:35.511092+0000","last_unstale":"2023-12-16T21:13:37.485644+0000","last_undegraded":"2023-12-16T21:13:37.485644+0000","last_fullsized":"2023-12-16T21:13:37.485644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485618+0000","last_change":"2023-12-16T21:13:36.494416+0000","last_active":"2023-12-16T21:13:37.485618+0000","last_peered":"2023-12-16T21:13:37.485618+0000","last_clean":"2023-12-16T21:13:37.485618+0000","last_became_active":"2023-12-16T21:13:36.494121+0000","last_became_peered":"2023-12-16T21:13:36.494121+0000","last_unstale":"2023-12-16T21:13:37.485618+0000","last_undegraded":"2023-12-16T21:13:37.485618+0000","last_fullsized":"2023-12-16T21:13:37.485618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968502+0000","last_change":"2023-12-16T21:13:35.510139+0000","last_active":"2023-12-16T21:13:37.968502+0000","last_peered":"2023-12-16T21:13:37.968502+0000","last_clean":"2023-12-16T21:13:37.968502+0000","last_became_active":"2023-12-16T21:13:35.509804+0000","last_became_peered":"2023-12-16T21:13:35.509804+0000","last_unstale":"2023-12-16T21:13:37.968502+0000","last_undegraded":"2023-12-16T21:13:37.968502+0000","last_fullsized":"2023-12-16T21:13:37.968502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:56:59.813907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:48.378 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:13:48.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.521+0000 7fc1b45be640 1 Processor -- start 2023-12-16T21:13:48.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.521+0000 7fc1b45be640 1 -- start start 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b45be640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1ac06ea50 0x7fc1ac06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b45be640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b45be640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac161d20 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b45be640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ac0d1ee0 con 0x7fc1ac06ea50 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b45be640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac06f370 con 0x7fc1ac160fb0 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b235a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1ac06ea50 0x7fc1ac06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b2b5b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc1ac0b3ff0 0x7fc1ac0b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53580/0 (socket says 172.21.15.46:53580) 2023-12-16T21:13:48.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b2b5b640 1 -- 172.21.15.46:0/3240549557 learned_addr learned my addr 172.21.15.46:0/3240549557 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b1b59640 1 --2- 172.21.15.46:0/3240549557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b235a640 1 -- 172.21.15.46:0/3240549557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 msgr2=0x7fc1ac161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.522+0000 7fc1b235a640 1 --2- 172.21.15.46:0/3240549557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b235a640 1 -- 172.21.15.46:0/3240549557 >> v1:172.21.15.46:6789/0 conn(0x7fc1ac0b3ff0 legacy=0x7fc1ac0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b235a640 1 -- 172.21.15.46:0/3240549557 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac15e760 con 0x7fc1ac06ea50 2023-12-16T21:13:48.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b1358640 1 -- 172.21.15.46:0/3240549557 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719599895 0 0) 0x7fc1ac161d20 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b235a640 1 --2- 172.21.15.46:0/3240549557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac06ee30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40097a0 tx=0x7fc1a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecefa2061d31ace8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b1358640 1 -- 172.21.15.46:0/3240549557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4031830 con 0x7fc1ac06ea50 2023-12-16T21:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b1358640 1 -- 172.21.15.46:0/3240549557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4031990 con 0x7fc1ac06ea50 2023-12-16T21:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.523+0000 7fc1b1358640 1 -- 172.21.15.46:0/3240549557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4031d20 con 0x7fc1ac06ea50 2023-12-16T21:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 -- 172.21.15.46:0/3240549557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac06ea50 msgr2=0x7fc1ac06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 --2- 172.21.15.46:0/3240549557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac06ee30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40097a0 tx=0x7fc1a402b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 -- 172.21.15.46:0/3240549557 shutdown_connections 2023-12-16T21:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 --2- 172.21.15.46:0/3240549557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 --2- 172.21.15.46:0/3240549557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac06ee30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 -- 172.21.15.46:0/3240549557 >> 172.21.15.46:0/3240549557 conn(0x7fc1ac06a6b0 msgr2=0x7fc1ac06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 -- 172.21.15.46:0/3240549557 shutdown_connections 2023-12-16T21:13:48.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.524+0000 7fc1b45be640 1 -- 172.21.15.46:0/3240549557 wait complete. 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.525+0000 7fc1b45be640 1 Processor -- start 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.525+0000 7fc1b45be640 1 -- start start 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b45be640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b45be640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 0x7fc1ac1e7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b45be640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac06b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b45be640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac154df0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b45be640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1e8260 con 0x7fc1ac06ea50 2023-12-16T21:13:48.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b45be640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1e8540 con 0x7fc1ac160fb0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 0x7fc1ac1e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 0x7fc1ac1e7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36500/0 (socket says 172.21.15.46:36500) 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 -- 172.21.15.46:0/2182427582 learned_addr learned my addr 172.21.15.46:0/2182427582 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b2b5b640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac06b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b235a640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 -- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 msgr2=0x7fc1ac06b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac06b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 -- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1ac06ea50 msgr2=0x7fc1ac1548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac1548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b1b59640 1 -- 172.21.15.46:0/2182427582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a4009400 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.526+0000 7fc1b235a640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.527+0000 7fc1b1b59640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 0x7fc1ac1e7860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000d800 tx=0x7fc1a000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.527+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000b6c0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.527+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a000b820 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.527+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a000bbb0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.527+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac1e8820 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.527+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1e8de0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.528+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac1e8fa0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.528+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc1a000bd10 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.529+0000 7fc1aaffd640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc19806ae40 0x7fc19806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.529+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1a0089af0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.529+0000 7fc1b235a640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc19806ae40 0x7fc19806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.530+0000 7fc1b235a640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc19806ae40 0x7fc19806d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4036000 tx=0x7fc1a403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.533+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc1a0054ba0 con 0x7fc1ac0b3ff0 2023-12-16T21:13:48.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.705+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc1ac15df60 con 0x7fc19806ae40 2023-12-16T21:13:48.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.713+0000 7fc1aaffd640 1 -- 172.21.15.46:0/2182427582 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264537 (secure 0 0 0) 0x7fc1a4044400 con 0x7fc19806ae40 2023-12-16T21:13:48.715 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:48.716 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc19806ae40 msgr2=0x7fc19806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc19806ae40 0x7fc19806d300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4036000 tx=0x7fc1a403c040 comp rx=0 tx=0).stop 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 msgr2=0x7fc1ac1e7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 0x7fc1ac1e7860 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000d800 tx=0x7fc1a000dcc0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 shutdown_connections 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc19806ae40 0x7fc19806d300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1ac0b3ff0 0x7fc1ac1e7860 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1ac160fb0 0x7fc1ac06b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 --2- 172.21.15.46:0/2182427582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1ac06ea50 0x7fc1ac1548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 >> 172.21.15.46:0/2182427582 conn(0x7fc1ac06a6b0 msgr2=0x7fc1ac0b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 shutdown_connections 2023-12-16T21:13:48.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.716+0000 7fc1b45be640 1 -- 172.21.15.46:0/2182427582 wait complete. 2023-12-16T21:13:48.726 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-12-16T21:13:47.957883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235752,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738968,"statfs":{"total":773094113280,"available":772852703232,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.474044"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580448+0000","last_change":"2023-12-16T21:13:36.497094+0000","last_active":"2023-12-16T21:13:37.580448+0000","last_peered":"2023-12-16T21:13:37.580448+0000","last_clean":"2023-12-16T21:13:37.580448+0000","last_became_active":"2023-12-16T21:13:36.496839+0000","last_became_peered":"2023-12-16T21:13:36.496839+0000","last_unstale":"2023-12-16T21:13:37.580448+0000","last_undegraded":"2023-12-16T21:13:37.580448+0000","last_fullsized":"2023-12-16T21:13:37.580448+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043043+0000","last_change":"2023-12-16T21:13:36.496641+0000","last_active":"2023-12-16T21:13:38.043043+0000","last_peered":"2023-12-16T21:13:38.043043+0000","last_clean":"2023-12-16T21:13:38.043043+0000","last_became_active":"2023-12-16T21:13:36.496301+0000","last_became_peered":"2023-12-16T21:13:36.496301+0000","last_unstale":"2023-12-16T21:13:38.043043+0000","last_undegraded":"2023-12-16T21:13:38.043043+0000","last_fullsized":"2023-12-16T21:13:38.043043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004757+0000","last_change":"2023-12-16T21:13:36.490436+0000","last_active":"2023-12-16T21:13:38.004757+0000","last_peered":"2023-12-16T21:13:38.004757+0000","last_clean":"2023-12-16T21:13:38.004757+0000","last_became_active":"2023-12-16T21:13:36.489881+0000","last_became_peered":"2023-12-16T21:13:36.489881+0000","last_unstale":"2023-12-16T21:13:38.004757+0000","last_undegraded":"2023-12-16T21:13:38.004757+0000","last_fullsized":"2023-12-16T21:13:38.004757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487045+0000","last_change":"2023-12-16T21:13:36.507326+0000","last_active":"2023-12-16T21:13:37.487045+0000","last_peered":"2023-12-16T21:13:37.487045+0000","last_clean":"2023-12-16T21:13:37.487045+0000","last_became_active":"2023-12-16T21:13:36.507063+0000","last_became_peered":"2023-12-16T21:13:36.507063+0000","last_unstale":"2023-12-16T21:13:37.487045+0000","last_undegraded":"2023-12-16T21:13:37.487045+0000","last_fullsized":"2023-12-16T21:13:37.487045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580048+0000","last_change":"2023-12-16T21:13:36.494596+0000","last_active":"2023-12-16T21:13:37.580048+0000","last_peered":"2023-12-16T21:13:37.580048+0000","last_clean":"2023-12-16T21:13:37.580048+0000","last_became_active":"2023-12-16T21:13:36.494238+0000","last_became_peered":"2023-12-16T21:13:36.494238+0000","last_unstale":"2023-12-16T21:13:37.580048+0000","last_undegraded":"2023-12-16T21:13:37.580048+0000","last_fullsized":"2023-12-16T21:13:37.580048+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486248+0000","last_change":"2023-12-16T21:13:36.505505+0000","last_active":"2023-12-16T21:13:37.486248+0000","last_peered":"2023-12-16T21:13:37.486248+0000","last_clean":"2023-12-16T21:13:37.486248+0000","last_became_active":"2023-12-16T21:13:36.505205+0000","last_became_peered":"2023-12-16T21:13:36.505205+0000","last_unstale":"2023-12-16T21:13:37.486248+0000","last_undegraded":"2023-12-16T21:13:37.486248+0000","last_fullsized":"2023-12-16T21:13:37.486248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579992+0000","last_change":"2023-12-16T21:13:36.502009+0000","last_active":"2023-12-16T21:13:37.579992+0000","last_peered":"2023-12-16T21:13:37.579992+0000","last_clean":"2023-12-16T21:13:37.579992+0000","last_became_active":"2023-12-16T21:13:36.501672+0000","last_became_peered":"2023-12-16T21:13:36.501672+0000","last_unstale":"2023-12-16T21:13:37.579992+0000","last_undegraded":"2023-12-16T21:13:37.579992+0000","last_fullsized":"2023-12-16T21:13:37.579992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.484971+0000","last_change":"2023-12-16T21:13:36.502551+0000","last_active":"2023-12-16T21:13:37.484971+0000","last_peered":"2023-12-16T21:13:37.484971+0000","last_clean":"2023-12-16T21:13:37.484971+0000","last_became_active":"2023-12-16T21:13:36.502214+0000","last_became_peered":"2023-12-16T21:13:36.502214+0000","last_unstale":"2023-12-16T21:13:37.484971+0000","last_undegraded":"2023-12-16T21:13:37.484971+0000","last_fullsized":"2023-12-16T21:13:37.484971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580474+0000","last_change":"2023-12-16T21:13:36.507546+0000","last_active":"2023-12-16T21:13:37.580474+0000","last_peered":"2023-12-16T21:13:37.580474+0000","last_clean":"2023-12-16T21:13:37.580474+0000","last_became_active":"2023-12-16T21:13:36.507253+0000","last_became_peered":"2023-12-16T21:13:36.507253+0000","last_unstale":"2023-12-16T21:13:37.580474+0000","last_undegraded":"2023-12-16T21:13:37.580474+0000","last_fullsized":"2023-12-16T21:13:37.580474+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:22:24.104289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485901+0000","last_change":"2023-12-16T21:13:36.508602+0000","last_active":"2023-12-16T21:13:37.485901+0000","last_peered":"2023-12-16T21:13:37.485901+0000","last_clean":"2023-12-16T21:13:37.485901+0000","last_became_active":"2023-12-16T21:13:36.508223+0000","last_became_peered":"2023-12-16T21:13:36.508223+0000","last_unstale":"2023-12-16T21:13:37.485901+0000","last_undegraded":"2023-12-16T21:13:37.485901+0000","last_fullsized":"2023-12-16T21:13:37.485901+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485806+0000","last_change":"2023-12-16T21:13:36.508624+0000","last_active":"2023-12-16T21:13:37.485806+0000","last_peered":"2023-12-16T21:13:37.485806+0000","last_clean":"2023-12-16T21:13:37.485806+0000","last_became_active":"2023-12-16T21:13:36.508293+0000","last_became_peered":"2023-12-16T21:13:36.508293+0000","last_unstale":"2023-12-16T21:13:37.485806+0000","last_undegraded":"2023-12-16T21:13:37.485806+0000","last_fullsized":"2023-12-16T21:13:37.485806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043125+0000","last_change":"2023-12-16T21:13:36.499195+0000","last_active":"2023-12-16T21:13:38.043125+0000","last_peered":"2023-12-16T21:13:38.043125+0000","last_clean":"2023-12-16T21:13:38.043125+0000","last_became_active":"2023-12-16T21:13:36.498927+0000","last_became_peered":"2023-12-16T21:13:36.498927+0000","last_unstale":"2023-12-16T21:13:38.043125+0000","last_undegraded":"2023-12-16T21:13:38.043125+0000","last_fullsized":"2023-12-16T21:13:38.043125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:20.146932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485287+0000","last_change":"2023-12-16T21:13:36.494278+0000","last_active":"2023-12-16T21:13:37.485287+0000","last_peered":"2023-12-16T21:13:37.485287+0000","last_clean":"2023-12-16T21:13:37.485287+0000","last_became_active":"2023-12-16T21:13:36.493970+0000","last_became_peered":"2023-12-16T21:13:36.493970+0000","last_unstale":"2023-12-16T21:13:37.485287+0000","last_undegraded":"2023-12-16T21:13:37.485287+0000","last_fullsized":"2023-12-16T21:13:37.485287+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004516+0000","last_change":"2023-12-16T21:13:36.499374+0000","last_active":"2023-12-16T21:13:38.004516+0000","last_peered":"2023-12-16T21:13:38.004516+0000","last_clean":"2023-12-16T21:13:38.004516+0000","last_became_active":"2023-12-16T21:13:36.499097+0000","last_became_peered":"2023-12-16T21:13:36.499097+0000","last_unstale":"2023-12-16T21:13:38.004516+0000","last_undegraded":"2023-12-16T21:13:38.004516+0000","last_fullsized":"2023-12-16T21:13:38.004516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:33.841187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042953+0000","last_change":"2023-12-16T21:13:36.504350+0000","last_active":"2023-12-16T21:13:38.042953+0000","last_peered":"2023-12-16T21:13:38.042953+0000","last_clean":"2023-12-16T21:13:38.042953+0000","last_became_active":"2023-12-16T21:13:36.504046+0000","last_became_peered":"2023-12-16T21:13:36.504046+0000","last_unstale":"2023-12-16T21:13:38.042953+0000","last_undegraded":"2023-12-16T21:13:38.042953+0000","last_fullsized":"2023-12-16T21:13:38.042953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485909+0000","last_change":"2023-12-16T21:13:36.502440+0000","last_active":"2023-12-16T21:13:37.485909+0000","last_peered":"2023-12-16T21:13:37.485909+0000","last_clean":"2023-12-16T21:13:37.485909+0000","last_became_active":"2023-12-16T21:13:36.502160+0000","last_became_peered":"2023-12-16T21:13:36.502160+0000","last_unstale":"2023-12-16T21:13:37.485909+0000","last_undegraded":"2023-12-16T21:13:37.485909+0000","last_fullsized":"2023-12-16T21:13:37.485909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487000+0000","last_change":"2023-12-16T21:13:36.508778+0000","last_active":"2023-12-16T21:13:37.487000+0000","last_peered":"2023-12-16T21:13:37.487000+0000","last_clean":"2023-12-16T21:13:37.487000+0000","last_became_active":"2023-12-16T21:13:36.508506+0000","last_became_peered":"2023-12-16T21:13:36.508506+0000","last_unstale":"2023-12-16T21:13:37.487000+0000","last_undegraded":"2023-12-16T21:13:37.487000+0000","last_fullsized":"2023-12-16T21:13:37.487000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042744+0000","last_change":"2023-12-16T21:13:36.494459+0000","last_active":"2023-12-16T21:13:38.042744+0000","last_peered":"2023-12-16T21:13:38.042744+0000","last_clean":"2023-12-16T21:13:38.042744+0000","last_became_active":"2023-12-16T21:13:36.494156+0000","last_became_peered":"2023-12-16T21:13:36.494156+0000","last_unstale":"2023-12-16T21:13:38.042744+0000","last_undegraded":"2023-12-16T21:13:38.042744+0000","last_fullsized":"2023-12-16T21:13:38.042744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:47.886346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580395+0000","last_change":"2023-12-16T21:13:36.496446+0000","last_active":"2023-12-16T21:13:37.580395+0000","last_peered":"2023-12-16T21:13:37.580395+0000","last_clean":"2023-12-16T21:13:37.580395+0000","last_became_active":"2023-12-16T21:13:36.496123+0000","last_became_peered":"2023-12-16T21:13:36.496123+0000","last_unstale":"2023-12-16T21:13:37.580395+0000","last_undegraded":"2023-12-16T21:13:37.580395+0000","last_fullsized":"2023-12-16T21:13:37.580395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:46:06.459817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486172+0000","last_change":"2023-12-16T21:13:36.505223+0000","last_active":"2023-12-16T21:13:37.486172+0000","last_peered":"2023-12-16T21:13:37.486172+0000","last_clean":"2023-12-16T21:13:37.486172+0000","last_became_active":"2023-12-16T21:13:36.504834+0000","last_became_peered":"2023-12-16T21:13:36.504834+0000","last_unstale":"2023-12-16T21:13:37.486172+0000","last_undegraded":"2023-12-16T21:13:37.486172+0000","last_fullsized":"2023-12-16T21:13:37.486172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004625+0000","last_change":"2023-12-16T21:13:36.500880+0000","last_active":"2023-12-16T21:13:38.004625+0000","last_peered":"2023-12-16T21:13:38.004625+0000","last_clean":"2023-12-16T21:13:38.004625+0000","last_became_active":"2023-12-16T21:13:36.500532+0000","last_became_peered":"2023-12-16T21:13:36.500532+0000","last_unstale":"2023-12-16T21:13:38.004625+0000","last_undegraded":"2023-12-16T21:13:38.004625+0000","last_fullsized":"2023-12-16T21:13:38.004625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:39.576958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486491+0000","last_change":"2023-12-16T21:13:36.507578+0000","last_active":"2023-12-16T21:13:37.486491+0000","last_peered":"2023-12-16T21:13:37.486491+0000","last_clean":"2023-12-16T21:13:37.486491+0000","last_became_active":"2023-12-16T21:13:36.507206+0000","last_became_peered":"2023-12-16T21:13:36.507206+0000","last_unstale":"2023-12-16T21:13:37.486491+0000","last_undegraded":"2023-12-16T21:13:37.486491+0000","last_fullsized":"2023-12-16T21:13:37.486491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485167+0000","last_change":"2023-12-16T21:13:36.506029+0000","last_active":"2023-12-16T21:13:37.485167+0000","last_peered":"2023-12-16T21:13:37.485167+0000","last_clean":"2023-12-16T21:13:37.485167+0000","last_became_active":"2023-12-16T21:13:36.505785+0000","last_became_peered":"2023-12-16T21:13:36.505785+0000","last_unstale":"2023-12-16T21:13:37.485167+0000","last_undegraded":"2023-12-16T21:13:37.485167+0000","last_fullsized":"2023-12-16T21:13:37.485167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486054+0000","last_change":"2023-12-16T21:13:36.504916+0000","last_active":"2023-12-16T21:13:37.486054+0000","last_peered":"2023-12-16T21:13:37.486054+0000","last_clean":"2023-12-16T21:13:37.486054+0000","last_became_active":"2023-12-16T21:13:36.504422+0000","last_became_peered":"2023-12-16T21:13:36.504422+0000","last_unstale":"2023-12-16T21:13:37.486054+0000","last_undegraded":"2023-12-16T21:13:37.486054+0000","last_fullsized":"2023-12-16T21:13:37.486054+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.503543+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.503210+0000","last_became_peered":"2023-12-16T21:13:36.503210+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486065+0000","last_change":"2023-12-16T21:13:36.494611+0000","last_active":"2023-12-16T21:13:37.486065+0000","last_peered":"2023-12-16T21:13:37.486065+0000","last_clean":"2023-12-16T21:13:37.486065+0000","last_became_active":"2023-12-16T21:13:36.494373+0000","last_became_peered":"2023-12-16T21:13:36.494373+0000","last_unstale":"2023-12-16T21:13:37.486065+0000","last_undegraded":"2023-12-16T21:13:37.486065+0000","last_fullsized":"2023-12-16T21:13:37.486065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043012+0000","last_change":"2023-12-16T21:13:36.498640+0000","last_active":"2023-12-16T21:13:38.043012+0000","last_peered":"2023-12-16T21:13:38.043012+0000","last_clean":"2023-12-16T21:13:38.043012+0000","last_became_active":"2023-12-16T21:13:36.498334+0000","last_became_peered":"2023-12-16T21:13:36.498334+0000","last_unstale":"2023-12-16T21:13:38.043012+0000","last_undegraded":"2023-12-16T21:13:38.043012+0000","last_fullsized":"2023-12-16T21:13:38.043012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485419+0000","last_change":"2023-12-16T21:13:36.506001+0000","last_active":"2023-12-16T21:13:37.485419+0000","last_peered":"2023-12-16T21:13:37.485419+0000","last_clean":"2023-12-16T21:13:37.485419+0000","last_became_active":"2023-12-16T21:13:36.505724+0000","last_became_peered":"2023-12-16T21:13:36.505724+0000","last_unstale":"2023-12-16T21:13:37.485419+0000","last_undegraded":"2023-12-16T21:13:37.485419+0000","last_fullsized":"2023-12-16T21:13:37.485419+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:14:41.865138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004718+0000","last_change":"2023-12-16T21:13:36.489974+0000","last_active":"2023-12-16T21:13:38.004718+0000","last_peered":"2023-12-16T21:13:38.004718+0000","last_clean":"2023-12-16T21:13:38.004718+0000","last_became_active":"2023-12-16T21:13:36.489597+0000","last_became_peered":"2023-12-16T21:13:36.489597+0000","last_unstale":"2023-12-16T21:13:38.004718+0000","last_undegraded":"2023-12-16T21:13:38.004718+0000","last_fullsized":"2023-12-16T21:13:38.004718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486570+0000","last_change":"2023-12-16T21:13:36.505535+0000","last_active":"2023-12-16T21:13:37.486570+0000","last_peered":"2023-12-16T21:13:37.486570+0000","last_clean":"2023-12-16T21:13:37.486570+0000","last_became_active":"2023-12-16T21:13:36.505185+0000","last_became_peered":"2023-12-16T21:13:36.505185+0000","last_unstale":"2023-12-16T21:13:37.486570+0000","last_undegraded":"2023-12-16T21:13:37.486570+0000","last_fullsized":"2023-12-16T21:13:37.486570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042731+0000","last_change":"2023-12-16T21:13:36.504923+0000","last_active":"2023-12-16T21:13:38.042731+0000","last_peered":"2023-12-16T21:13:38.042731+0000","last_clean":"2023-12-16T21:13:38.042731+0000","last_became_active":"2023-12-16T21:13:36.504451+0000","last_became_peered":"2023-12-16T21:13:36.504451+0000","last_unstale":"2023-12-16T21:13:38.042731+0000","last_undegraded":"2023-12-16T21:13:38.042731+0000","last_fullsized":"2023-12-16T21:13:38.042731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485417+0000","last_change":"2023-12-16T21:13:36.497205+0000","last_active":"2023-12-16T21:13:37.485417+0000","last_peered":"2023-12-16T21:13:37.485417+0000","last_clean":"2023-12-16T21:13:37.485417+0000","last_became_active":"2023-12-16T21:13:36.496919+0000","last_became_peered":"2023-12-16T21:13:36.496919+0000","last_unstale":"2023-12-16T21:13:37.485417+0000","last_undegraded":"2023-12-16T21:13:37.485417+0000","last_fullsized":"2023-12-16T21:13:37.485417+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042855+0000","last_change":"2023-12-16T21:13:36.503295+0000","last_active":"2023-12-16T21:13:38.042855+0000","last_peered":"2023-12-16T21:13:38.042855+0000","last_clean":"2023-12-16T21:13:38.042855+0000","last_became_active":"2023-12-16T21:13:36.502730+0000","last_became_peered":"2023-12-16T21:13:36.502730+0000","last_unstale":"2023-12-16T21:13:38.042855+0000","last_undegraded":"2023-12-16T21:13:38.042855+0000","last_fullsized":"2023-12-16T21:13:38.042855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:36.196785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485372+0000","last_change":"2023-12-16T21:13:35.508997+0000","last_active":"2023-12-16T21:13:37.485372+0000","last_peered":"2023-12-16T21:13:37.485372+0000","last_clean":"2023-12-16T21:13:37.485372+0000","last_became_active":"2023-12-16T21:13:35.508689+0000","last_became_peered":"2023-12-16T21:13:35.508689+0000","last_unstale":"2023-12-16T21:13:37.485372+0000","last_undegraded":"2023-12-16T21:13:37.485372+0000","last_fullsized":"2023-12-16T21:13:37.485372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004783+0000","last_change":"2023-12-16T21:13:36.504100+0000","last_active":"2023-12-16T21:13:38.004783+0000","last_peered":"2023-12-16T21:13:38.004783+0000","last_clean":"2023-12-16T21:13:38.004783+0000","last_became_active":"2023-12-16T21:13:36.503340+0000","last_became_peered":"2023-12-16T21:13:36.503340+0000","last_unstale":"2023-12-16T21:13:38.004783+0000","last_undegraded":"2023-12-16T21:13:38.004783+0000","last_fullsized":"2023-12-16T21:13:38.004783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485542+0000","last_change":"2023-12-16T21:13:35.508135+0000","last_active":"2023-12-16T21:13:37.485542+0000","last_peered":"2023-12-16T21:13:37.485542+0000","last_clean":"2023-12-16T21:13:37.485542+0000","last_became_active":"2023-12-16T21:13:35.507149+0000","last_became_peered":"2023-12-16T21:13:35.507149+0000","last_unstale":"2023-12-16T21:13:37.485542+0000","last_undegraded":"2023-12-16T21:13:37.485542+0000","last_fullsized":"2023-12-16T21:13:37.485542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485144+0000","last_change":"2023-12-16T21:13:35.510923+0000","last_active":"2023-12-16T21:13:37.485144+0000","last_peered":"2023-12-16T21:13:37.485144+0000","last_clean":"2023-12-16T21:13:37.485144+0000","last_became_active":"2023-12-16T21:13:35.510638+0000","last_became_peered":"2023-12-16T21:13:35.510638+0000","last_unstale":"2023-12-16T21:13:37.485144+0000","last_undegraded":"2023-12-16T21:13:37.485144+0000","last_fullsized":"2023-12-16T21:13:37.485144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:49:52.127676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485039+0000","last_change":"2023-12-16T21:13:36.490652+0000","last_active":"2023-12-16T21:13:37.485039+0000","last_peered":"2023-12-16T21:13:37.485039+0000","last_clean":"2023-12-16T21:13:37.485039+0000","last_became_active":"2023-12-16T21:13:36.489990+0000","last_became_peered":"2023-12-16T21:13:36.489990+0000","last_unstale":"2023-12-16T21:13:37.485039+0000","last_undegraded":"2023-12-16T21:13:37.485039+0000","last_fullsized":"2023-12-16T21:13:37.485039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:12.668826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486149+0000","last_change":"2023-12-16T21:13:35.498392+0000","last_active":"2023-12-16T21:13:37.486149+0000","last_peered":"2023-12-16T21:13:37.486149+0000","last_clean":"2023-12-16T21:13:37.486149+0000","last_became_active":"2023-12-16T21:13:35.498119+0000","last_became_peered":"2023-12-16T21:13:35.498119+0000","last_unstale":"2023-12-16T21:13:37.486149+0000","last_undegraded":"2023-12-16T21:13:37.486149+0000","last_fullsized":"2023-12-16T21:13:37.486149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042709+0000","last_change":"2023-12-16T21:13:36.499009+0000","last_active":"2023-12-16T21:13:38.042709+0000","last_peered":"2023-12-16T21:13:38.042709+0000","last_clean":"2023-12-16T21:13:38.042709+0000","last_became_active":"2023-12-16T21:13:36.498714+0000","last_became_peered":"2023-12-16T21:13:36.498714+0000","last_unstale":"2023-12-16T21:13:38.042709+0000","last_undegraded":"2023-12-16T21:13:38.042709+0000","last_fullsized":"2023-12-16T21:13:38.042709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485912+0000","last_change":"2023-12-16T21:13:35.508895+0000","last_active":"2023-12-16T21:13:37.485912+0000","last_peered":"2023-12-16T21:13:37.485912+0000","last_clean":"2023-12-16T21:13:37.485912+0000","last_became_active":"2023-12-16T21:13:35.508644+0000","last_became_peered":"2023-12-16T21:13:35.508644+0000","last_unstale":"2023-12-16T21:13:37.485912+0000","last_undegraded":"2023-12-16T21:13:37.485912+0000","last_fullsized":"2023-12-16T21:13:37.485912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486627+0000","last_change":"2023-12-16T21:13:36.508270+0000","last_active":"2023-12-16T21:13:37.486627+0000","last_peered":"2023-12-16T21:13:37.486627+0000","last_clean":"2023-12-16T21:13:37.486627+0000","last_became_active":"2023-12-16T21:13:36.507998+0000","last_became_peered":"2023-12-16T21:13:36.507998+0000","last_unstale":"2023-12-16T21:13:37.486627+0000","last_undegraded":"2023-12-16T21:13:37.486627+0000","last_fullsized":"2023-12-16T21:13:37.486627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486639+0000","last_change":"2023-12-16T21:13:35.508588+0000","last_active":"2023-12-16T21:13:37.486639+0000","last_peered":"2023-12-16T21:13:37.486639+0000","last_clean":"2023-12-16T21:13:37.486639+0000","last_became_active":"2023-12-16T21:13:35.508246+0000","last_became_peered":"2023-12-16T21:13:35.508246+0000","last_unstale":"2023-12-16T21:13:37.486639+0000","last_undegraded":"2023-12-16T21:13:37.486639+0000","last_fullsized":"2023-12-16T21:13:37.486639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004928+0000","last_change":"2023-12-16T21:13:36.507570+0000","last_active":"2023-12-16T21:13:38.004928+0000","last_peered":"2023-12-16T21:13:38.004928+0000","last_clean":"2023-12-16T21:13:38.004928+0000","last_became_active":"2023-12-16T21:13:36.507226+0000","last_became_peered":"2023-12-16T21:13:36.507226+0000","last_unstale":"2023-12-16T21:13:38.004928+0000","last_undegraded":"2023-12-16T21:13:38.004928+0000","last_fullsized":"2023-12-16T21:13:38.004928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968370+0000","last_change":"2023-12-16T21:13:35.509646+0000","last_active":"2023-12-16T21:13:37.968370+0000","last_peered":"2023-12-16T21:13:37.968370+0000","last_clean":"2023-12-16T21:13:37.968370+0000","last_became_active":"2023-12-16T21:13:35.509335+0000","last_became_peered":"2023-12-16T21:13:35.509335+0000","last_unstale":"2023-12-16T21:13:37.968370+0000","last_undegraded":"2023-12-16T21:13:37.968370+0000","last_fullsized":"2023-12-16T21:13:37.968370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043143+0000","last_change":"2023-12-16T21:13:36.499896+0000","last_active":"2023-12-16T21:13:38.043143+0000","last_peered":"2023-12-16T21:13:38.043143+0000","last_clean":"2023-12-16T21:13:38.043143+0000","last_became_active":"2023-12-16T21:13:36.499625+0000","last_became_peered":"2023-12-16T21:13:36.499625+0000","last_unstale":"2023-12-16T21:13:38.043143+0000","last_undegraded":"2023-12-16T21:13:38.043143+0000","last_fullsized":"2023-12-16T21:13:38.043143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:50.273412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486051+0000","last_change":"2023-12-16T21:13:35.507588+0000","last_active":"2023-12-16T21:13:37.486051+0000","last_peered":"2023-12-16T21:13:37.486051+0000","last_clean":"2023-12-16T21:13:37.486051+0000","last_became_active":"2023-12-16T21:13:35.507264+0000","last_became_peered":"2023-12-16T21:13:35.507264+0000","last_unstale":"2023-12-16T21:13:37.486051+0000","last_undegraded":"2023-12-16T21:13:37.486051+0000","last_fullsized":"2023-12-16T21:13:37.486051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580181+0000","last_change":"2023-12-16T21:13:36.495697+0000","last_active":"2023-12-16T21:13:37.580181+0000","last_peered":"2023-12-16T21:13:37.580181+0000","last_clean":"2023-12-16T21:13:37.580181+0000","last_became_active":"2023-12-16T21:13:36.495099+0000","last_became_peered":"2023-12-16T21:13:36.495099+0000","last_unstale":"2023-12-16T21:13:37.580181+0000","last_undegraded":"2023-12-16T21:13:37.580181+0000","last_fullsized":"2023-12-16T21:13:37.580181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968435+0000","last_change":"2023-12-16T21:13:35.502116+0000","last_active":"2023-12-16T21:13:37.968435+0000","last_peered":"2023-12-16T21:13:37.968435+0000","last_clean":"2023-12-16T21:13:37.968435+0000","last_became_active":"2023-12-16T21:13:35.501814+0000","last_became_peered":"2023-12-16T21:13:35.501814+0000","last_unstale":"2023-12-16T21:13:37.968435+0000","last_undegraded":"2023-12-16T21:13:37.968435+0000","last_fullsized":"2023-12-16T21:13:37.968435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004596+0000","last_change":"2023-12-16T21:13:36.499212+0000","last_active":"2023-12-16T21:13:38.004596+0000","last_peered":"2023-12-16T21:13:38.004596+0000","last_clean":"2023-12-16T21:13:38.004596+0000","last_became_active":"2023-12-16T21:13:36.498745+0000","last_became_peered":"2023-12-16T21:13:36.498745+0000","last_unstale":"2023-12-16T21:13:38.004596+0000","last_undegraded":"2023-12-16T21:13:38.004596+0000","last_fullsized":"2023-12-16T21:13:38.004596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:10.671619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486589+0000","last_change":"2023-12-16T21:13:35.506529+0000","last_active":"2023-12-16T21:13:37.486589+0000","last_peered":"2023-12-16T21:13:37.486589+0000","last_clean":"2023-12-16T21:13:37.486589+0000","last_became_active":"2023-12-16T21:13:35.506231+0000","last_became_peered":"2023-12-16T21:13:35.506231+0000","last_unstale":"2023-12-16T21:13:37.486589+0000","last_undegraded":"2023-12-16T21:13:37.486589+0000","last_fullsized":"2023-12-16T21:13:37.486589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580108+0000","last_change":"2023-12-16T21:13:36.504072+0000","last_active":"2023-12-16T21:13:37.580108+0000","last_peered":"2023-12-16T21:13:37.580108+0000","last_clean":"2023-12-16T21:13:37.580108+0000","last_became_active":"2023-12-16T21:13:36.503801+0000","last_became_peered":"2023-12-16T21:13:36.503801+0000","last_unstale":"2023-12-16T21:13:37.580108+0000","last_undegraded":"2023-12-16T21:13:37.580108+0000","last_fullsized":"2023-12-16T21:13:37.580108+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485664+0000","last_change":"2023-12-16T21:13:35.508204+0000","last_active":"2023-12-16T21:13:37.485664+0000","last_peered":"2023-12-16T21:13:37.485664+0000","last_clean":"2023-12-16T21:13:37.485664+0000","last_became_active":"2023-12-16T21:13:35.507929+0000","last_became_peered":"2023-12-16T21:13:35.507929+0000","last_unstale":"2023-12-16T21:13:37.485664+0000","last_undegraded":"2023-12-16T21:13:37.485664+0000","last_fullsized":"2023-12-16T21:13:37.485664+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485428+0000","last_change":"2023-12-16T21:13:36.497155+0000","last_active":"2023-12-16T21:13:37.485428+0000","last_peered":"2023-12-16T21:13:37.485428+0000","last_clean":"2023-12-16T21:13:37.485428+0000","last_became_active":"2023-12-16T21:13:36.496862+0000","last_became_peered":"2023-12-16T21:13:36.496862+0000","last_unstale":"2023-12-16T21:13:37.485428+0000","last_undegraded":"2023-12-16T21:13:37.485428+0000","last_fullsized":"2023-12-16T21:13:37.485428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968533+0000","last_change":"2023-12-16T21:13:35.508118+0000","last_active":"2023-12-16T21:13:37.968533+0000","last_peered":"2023-12-16T21:13:37.968533+0000","last_clean":"2023-12-16T21:13:37.968533+0000","last_became_active":"2023-12-16T21:13:35.507827+0000","last_became_peered":"2023-12-16T21:13:35.507827+0000","last_unstale":"2023-12-16T21:13:37.968533+0000","last_undegraded":"2023-12-16T21:13:37.968533+0000","last_fullsized":"2023-12-16T21:13:37.968533+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:21.335163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580055+0000","last_change":"2023-12-16T21:13:36.504720+0000","last_active":"2023-12-16T21:13:37.580055+0000","last_peered":"2023-12-16T21:13:37.580055+0000","last_clean":"2023-12-16T21:13:37.580055+0000","last_became_active":"2023-12-16T21:13:36.504444+0000","last_became_peered":"2023-12-16T21:13:36.504444+0000","last_unstale":"2023-12-16T21:13:37.580055+0000","last_undegraded":"2023-12-16T21:13:37.580055+0000","last_fullsized":"2023-12-16T21:13:37.580055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T09:11:58.118666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968696+0000","last_change":"2023-12-16T21:13:35.509590+0000","last_active":"2023-12-16T21:13:37.968696+0000","last_peered":"2023-12-16T21:13:37.968696+0000","last_clean":"2023-12-16T21:13:37.968696+0000","last_became_active":"2023-12-16T21:13:35.509278+0000","last_became_peered":"2023-12-16T21:13:35.509278+0000","last_unstale":"2023-12-16T21:13:37.968696+0000","last_undegraded":"2023-12-16T21:13:37.968696+0000","last_fullsized":"2023-12-16T21:13:37.968696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:46.659974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580433+0000","last_change":"2023-12-16T21:13:36.498151+0000","last_active":"2023-12-16T21:13:37.580433+0000","last_peered":"2023-12-16T21:13:37.580433+0000","last_clean":"2023-12-16T21:13:37.580433+0000","last_became_active":"2023-12-16T21:13:36.497907+0000","last_became_peered":"2023-12-16T21:13:36.497907+0000","last_unstale":"2023-12-16T21:13:37.580433+0000","last_undegraded":"2023-12-16T21:13:37.580433+0000","last_fullsized":"2023-12-16T21:13:37.580433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968604+0000","last_change":"2023-12-16T21:13:35.507707+0000","last_active":"2023-12-16T21:13:37.968604+0000","last_peered":"2023-12-16T21:13:37.968604+0000","last_clean":"2023-12-16T21:13:37.968604+0000","last_became_active":"2023-12-16T21:13:35.507148+0000","last_became_peered":"2023-12-16T21:13:35.507148+0000","last_unstale":"2023-12-16T21:13:37.968604+0000","last_undegraded":"2023-12-16T21:13:37.968604+0000","last_fullsized":"2023-12-16T21:13:37.968604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:36:35.192870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580539+0000","last_change":"2023-12-16T21:13:36.503790+0000","last_active":"2023-12-16T21:13:37.580539+0000","last_peered":"2023-12-16T21:13:37.580539+0000","last_clean":"2023-12-16T21:13:37.580539+0000","last_became_active":"2023-12-16T21:13:36.503514+0000","last_became_peered":"2023-12-16T21:13:36.503514+0000","last_unstale":"2023-12-16T21:13:37.580539+0000","last_undegraded":"2023-12-16T21:13:37.580539+0000","last_fullsized":"2023-12-16T21:13:37.580539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:43:28.130579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968380+0000","last_change":"2023-12-16T21:13:35.507575+0000","last_active":"2023-12-16T21:13:37.968380+0000","last_peered":"2023-12-16T21:13:37.968380+0000","last_clean":"2023-12-16T21:13:37.968380+0000","last_became_active":"2023-12-16T21:13:35.507291+0000","last_became_peered":"2023-12-16T21:13:35.507291+0000","last_unstale":"2023-12-16T21:13:37.968380+0000","last_undegraded":"2023-12-16T21:13:37.968380+0000","last_fullsized":"2023-12-16T21:13:37.968380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:07:09.667704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487138+0000","last_change":"2023-12-16T21:13:36.507654+0000","last_active":"2023-12-16T21:13:37.487138+0000","last_peered":"2023-12-16T21:13:37.487138+0000","last_clean":"2023-12-16T21:13:37.487138+0000","last_became_active":"2023-12-16T21:13:36.507245+0000","last_became_peered":"2023-12-16T21:13:36.507245+0000","last_unstale":"2023-12-16T21:13:37.487138+0000","last_undegraded":"2023-12-16T21:13:37.487138+0000","last_fullsized":"2023-12-16T21:13:37.487138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:20:17.781158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968585+0000","last_change":"2023-12-16T21:13:35.509065+0000","last_active":"2023-12-16T21:13:37.968585+0000","last_peered":"2023-12-16T21:13:37.968585+0000","last_clean":"2023-12-16T21:13:37.968585+0000","last_became_active":"2023-12-16T21:13:35.508808+0000","last_became_peered":"2023-12-16T21:13:35.508808+0000","last_unstale":"2023-12-16T21:13:37.968585+0000","last_undegraded":"2023-12-16T21:13:37.968585+0000","last_fullsized":"2023-12-16T21:13:37.968585+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580284+0000","last_change":"2023-12-16T21:13:36.507427+0000","last_active":"2023-12-16T21:13:37.580284+0000","last_peered":"2023-12-16T21:13:37.580284+0000","last_clean":"2023-12-16T21:13:37.580284+0000","last_became_active":"2023-12-16T21:13:36.507170+0000","last_became_peered":"2023-12-16T21:13:36.507170+0000","last_unstale":"2023-12-16T21:13:37.580284+0000","last_undegraded":"2023-12-16T21:13:37.580284+0000","last_fullsized":"2023-12-16T21:13:37.580284+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:18.365127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:36.490630+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:13:36.490630+0000","last_peered":"2023-12-16T21:13:36.490630+0000","last_clean":"2023-12-16T21:13:36.490630+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:13:36.490630+0000","last_undegraded":"2023-12-16T21:13:36.490630+0000","last_fullsized":"2023-12-16T21:13:36.490630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486371+0000","last_change":"2023-12-16T21:13:35.505945+0000","last_active":"2023-12-16T21:13:37.486371+0000","last_peered":"2023-12-16T21:13:37.486371+0000","last_clean":"2023-12-16T21:13:37.486371+0000","last_became_active":"2023-12-16T21:13:35.505670+0000","last_became_peered":"2023-12-16T21:13:35.505670+0000","last_unstale":"2023-12-16T21:13:37.486371+0000","last_undegraded":"2023-12-16T21:13:37.486371+0000","last_fullsized":"2023-12-16T21:13:37.486371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485341+0000","last_change":"2023-12-16T21:13:36.507596+0000","last_active":"2023-12-16T21:13:37.485341+0000","last_peered":"2023-12-16T21:13:37.485341+0000","last_clean":"2023-12-16T21:13:37.485341+0000","last_became_active":"2023-12-16T21:13:36.507302+0000","last_became_peered":"2023-12-16T21:13:36.507302+0000","last_unstale":"2023-12-16T21:13:37.485341+0000","last_undegraded":"2023-12-16T21:13:37.485341+0000","last_fullsized":"2023-12-16T21:13:37.485341+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579838+0000","last_change":"2023-12-16T21:13:35.512856+0000","last_active":"2023-12-16T21:13:37.579838+0000","last_peered":"2023-12-16T21:13:37.579838+0000","last_clean":"2023-12-16T21:13:37.579838+0000","last_became_active":"2023-12-16T21:13:35.512539+0000","last_became_peered":"2023-12-16T21:13:35.512539+0000","last_unstale":"2023-12-16T21:13:37.579838+0000","last_undegraded":"2023-12-16T21:13:37.579838+0000","last_fullsized":"2023-12-16T21:13:37.579838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:00:19.143018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004529+0000","last_change":"2023-12-16T21:13:36.503724+0000","last_active":"2023-12-16T21:13:38.004529+0000","last_peered":"2023-12-16T21:13:38.004529+0000","last_clean":"2023-12-16T21:13:38.004529+0000","last_became_active":"2023-12-16T21:13:36.503392+0000","last_became_peered":"2023-12-16T21:13:36.503392+0000","last_unstale":"2023-12-16T21:13:38.004529+0000","last_undegraded":"2023-12-16T21:13:38.004529+0000","last_fullsized":"2023-12-16T21:13:38.004529+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:02.640523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968573+0000","last_change":"2023-12-16T21:13:35.506777+0000","last_active":"2023-12-16T21:13:37.968573+0000","last_peered":"2023-12-16T21:13:37.968573+0000","last_clean":"2023-12-16T21:13:37.968573+0000","last_became_active":"2023-12-16T21:13:35.506483+0000","last_became_peered":"2023-12-16T21:13:35.506483+0000","last_unstale":"2023-12-16T21:13:37.968573+0000","last_undegraded":"2023-12-16T21:13:37.968573+0000","last_fullsized":"2023-12-16T21:13:37.968573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:00:25.556208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.042790+0000","last_change":"2023-12-16T21:13:36.496004+0000","last_active":"2023-12-16T21:13:38.042790+0000","last_peered":"2023-12-16T21:13:38.042790+0000","last_clean":"2023-12-16T21:13:38.042790+0000","last_became_active":"2023-12-16T21:13:36.495712+0000","last_became_peered":"2023-12-16T21:13:36.495712+0000","last_unstale":"2023-12-16T21:13:38.042790+0000","last_undegraded":"2023-12-16T21:13:38.042790+0000","last_fullsized":"2023-12-16T21:13:38.042790+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486820+0000","last_change":"2023-12-16T21:13:35.507557+0000","last_active":"2023-12-16T21:13:37.486820+0000","last_peered":"2023-12-16T21:13:37.486820+0000","last_clean":"2023-12-16T21:13:37.486820+0000","last_became_active":"2023-12-16T21:13:35.507175+0000","last_became_peered":"2023-12-16T21:13:35.507175+0000","last_unstale":"2023-12-16T21:13:37.486820+0000","last_undegraded":"2023-12-16T21:13:37.486820+0000","last_fullsized":"2023-12-16T21:13:37.486820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:21:11.664911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043017+0000","last_change":"2023-12-16T21:13:36.502502+0000","last_active":"2023-12-16T21:13:38.043017+0000","last_peered":"2023-12-16T21:13:38.043017+0000","last_clean":"2023-12-16T21:13:38.043017+0000","last_became_active":"2023-12-16T21:13:36.501893+0000","last_became_peered":"2023-12-16T21:13:36.501893+0000","last_unstale":"2023-12-16T21:13:38.043017+0000","last_undegraded":"2023-12-16T21:13:38.043017+0000","last_fullsized":"2023-12-16T21:13:38.043017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:46.012193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968352+0000","last_change":"2023-12-16T21:13:35.513556+0000","last_active":"2023-12-16T21:13:37.968352+0000","last_peered":"2023-12-16T21:13:37.968352+0000","last_clean":"2023-12-16T21:13:37.968352+0000","last_became_active":"2023-12-16T21:13:35.513208+0000","last_became_peered":"2023-12-16T21:13:35.513208+0000","last_unstale":"2023-12-16T21:13:37.968352+0000","last_undegraded":"2023-12-16T21:13:37.968352+0000","last_fullsized":"2023-12-16T21:13:37.968352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486956+0000","last_change":"2023-12-16T21:13:36.504553+0000","last_active":"2023-12-16T21:13:37.486956+0000","last_peered":"2023-12-16T21:13:37.486956+0000","last_clean":"2023-12-16T21:13:37.486956+0000","last_became_active":"2023-12-16T21:13:36.504276+0000","last_became_peered":"2023-12-16T21:13:36.504276+0000","last_unstale":"2023-12-16T21:13:37.486956+0000","last_undegraded":"2023-12-16T21:13:37.486956+0000","last_fullsized":"2023-12-16T21:13:37.486956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:34.609561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580388+0000","last_change":"2023-12-16T21:13:35.511628+0000","last_active":"2023-12-16T21:13:37.580388+0000","last_peered":"2023-12-16T21:13:37.580388+0000","last_clean":"2023-12-16T21:13:37.580388+0000","last_became_active":"2023-12-16T21:13:35.510934+0000","last_became_peered":"2023-12-16T21:13:35.510934+0000","last_unstale":"2023-12-16T21:13:37.580388+0000","last_undegraded":"2023-12-16T21:13:37.580388+0000","last_fullsized":"2023-12-16T21:13:37.580388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:21.664872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043123+0000","last_change":"2023-12-16T21:13:36.495330+0000","last_active":"2023-12-16T21:13:38.043123+0000","last_peered":"2023-12-16T21:13:38.043123+0000","last_clean":"2023-12-16T21:13:38.043123+0000","last_became_active":"2023-12-16T21:13:36.495064+0000","last_became_peered":"2023-12-16T21:13:36.495064+0000","last_unstale":"2023-12-16T21:13:38.043123+0000","last_undegraded":"2023-12-16T21:13:38.043123+0000","last_fullsized":"2023-12-16T21:13:38.043123+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043073+0000","last_change":"2023-12-16T21:13:35.508842+0000","last_active":"2023-12-16T21:13:38.043073+0000","last_peered":"2023-12-16T21:13:38.043073+0000","last_clean":"2023-12-16T21:13:38.043073+0000","last_became_active":"2023-12-16T21:13:35.508552+0000","last_became_peered":"2023-12-16T21:13:35.508552+0000","last_unstale":"2023-12-16T21:13:38.043073+0000","last_undegraded":"2023-12-16T21:13:38.043073+0000","last_fullsized":"2023-12-16T21:13:38.043073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485663+0000","last_change":"2023-12-16T21:13:36.497136+0000","last_active":"2023-12-16T21:13:37.485663+0000","last_peered":"2023-12-16T21:13:37.485663+0000","last_clean":"2023-12-16T21:13:37.485663+0000","last_became_active":"2023-12-16T21:13:36.496859+0000","last_became_peered":"2023-12-16T21:13:36.496859+0000","last_unstale":"2023-12-16T21:13:37.485663+0000","last_undegraded":"2023-12-16T21:13:37.485663+0000","last_fullsized":"2023-12-16T21:13:37.485663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486007+0000","last_change":"2023-12-16T21:13:35.512677+0000","last_active":"2023-12-16T21:13:37.486007+0000","last_peered":"2023-12-16T21:13:37.486007+0000","last_clean":"2023-12-16T21:13:37.486007+0000","last_became_active":"2023-12-16T21:13:35.511936+0000","last_became_peered":"2023-12-16T21:13:35.511936+0000","last_unstale":"2023-12-16T21:13:37.486007+0000","last_undegraded":"2023-12-16T21:13:37.486007+0000","last_fullsized":"2023-12-16T21:13:37.486007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:11:46.882431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004860+0000","last_change":"2023-12-16T21:13:36.499822+0000","last_active":"2023-12-16T21:13:38.004860+0000","last_peered":"2023-12-16T21:13:38.004860+0000","last_clean":"2023-12-16T21:13:38.004860+0000","last_became_active":"2023-12-16T21:13:36.499517+0000","last_became_peered":"2023-12-16T21:13:36.499517+0000","last_unstale":"2023-12-16T21:13:38.004860+0000","last_undegraded":"2023-12-16T21:13:38.004860+0000","last_fullsized":"2023-12-16T21:13:38.004860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:26.711470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.579837+0000","last_change":"2023-12-16T21:13:35.511783+0000","last_active":"2023-12-16T21:13:37.579837+0000","last_peered":"2023-12-16T21:13:37.579837+0000","last_clean":"2023-12-16T21:13:37.579837+0000","last_became_active":"2023-12-16T21:13:35.510752+0000","last_became_peered":"2023-12-16T21:13:35.510752+0000","last_unstale":"2023-12-16T21:13:37.579837+0000","last_undegraded":"2023-12-16T21:13:37.579837+0000","last_fullsized":"2023-12-16T21:13:37.579837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:33:45.008279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.043007+0000","last_change":"2023-12-16T21:13:36.503029+0000","last_active":"2023-12-16T21:13:38.043007+0000","last_peered":"2023-12-16T21:13:38.043007+0000","last_clean":"2023-12-16T21:13:38.043007+0000","last_became_active":"2023-12-16T21:13:36.502641+0000","last_became_peered":"2023-12-16T21:13:36.502641+0000","last_unstale":"2023-12-16T21:13:38.043007+0000","last_undegraded":"2023-12-16T21:13:38.043007+0000","last_fullsized":"2023-12-16T21:13:38.043007+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:22.339077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968311+0000","last_change":"2023-12-16T21:13:35.511492+0000","last_active":"2023-12-16T21:13:37.968311+0000","last_peered":"2023-12-16T21:13:37.968311+0000","last_clean":"2023-12-16T21:13:37.968311+0000","last_became_active":"2023-12-16T21:13:35.511118+0000","last_became_peered":"2023-12-16T21:13:35.511118+0000","last_unstale":"2023-12-16T21:13:37.968311+0000","last_undegraded":"2023-12-16T21:13:37.968311+0000","last_fullsized":"2023-12-16T21:13:37.968311+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:17:38.573044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485112+0000","last_change":"2023-12-16T21:13:36.495196+0000","last_active":"2023-12-16T21:13:37.485112+0000","last_peered":"2023-12-16T21:13:37.485112+0000","last_clean":"2023-12-16T21:13:37.485112+0000","last_became_active":"2023-12-16T21:13:36.494933+0000","last_became_peered":"2023-12-16T21:13:36.494933+0000","last_unstale":"2023-12-16T21:13:37.485112+0000","last_undegraded":"2023-12-16T21:13:37.485112+0000","last_fullsized":"2023-12-16T21:13:37.485112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:20.362346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968321+0000","last_change":"2023-12-16T21:13:35.506652+0000","last_active":"2023-12-16T21:13:37.968321+0000","last_peered":"2023-12-16T21:13:37.968321+0000","last_clean":"2023-12-16T21:13:37.968321+0000","last_became_active":"2023-12-16T21:13:35.506349+0000","last_became_peered":"2023-12-16T21:13:35.506349+0000","last_unstale":"2023-12-16T21:13:37.968321+0000","last_undegraded":"2023-12-16T21:13:37.968321+0000","last_fullsized":"2023-12-16T21:13:37.968321+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:43:01.636608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485338+0000","last_change":"2023-12-16T21:13:36.490929+0000","last_active":"2023-12-16T21:13:37.485338+0000","last_peered":"2023-12-16T21:13:37.485338+0000","last_clean":"2023-12-16T21:13:37.485338+0000","last_became_active":"2023-12-16T21:13:36.490557+0000","last_became_peered":"2023-12-16T21:13:36.490557+0000","last_unstale":"2023-12-16T21:13:37.485338+0000","last_undegraded":"2023-12-16T21:13:37.485338+0000","last_fullsized":"2023-12-16T21:13:37.485338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:03:36.947119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580200+0000","last_change":"2023-12-16T21:13:35.511062+0000","last_active":"2023-12-16T21:13:37.580200+0000","last_peered":"2023-12-16T21:13:37.580200+0000","last_clean":"2023-12-16T21:13:37.580200+0000","last_became_active":"2023-12-16T21:13:35.510780+0000","last_became_peered":"2023-12-16T21:13:35.510780+0000","last_unstale":"2023-12-16T21:13:37.580200+0000","last_undegraded":"2023-12-16T21:13:37.580200+0000","last_fullsized":"2023-12-16T21:13:37.580200+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:25.707555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485439+0000","last_change":"2023-12-16T21:13:36.499546+0000","last_active":"2023-12-16T21:13:37.485439+0000","last_peered":"2023-12-16T21:13:37.485439+0000","last_clean":"2023-12-16T21:13:37.485439+0000","last_became_active":"2023-12-16T21:13:36.499236+0000","last_became_peered":"2023-12-16T21:13:36.499236+0000","last_unstale":"2023-12-16T21:13:37.485439+0000","last_undegraded":"2023-12-16T21:13:37.485439+0000","last_fullsized":"2023-12-16T21:13:37.485439+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:25:13.328816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.486745+0000","last_change":"2023-12-16T21:13:36.512546+0000","last_active":"2023-12-16T21:13:37.486745+0000","last_peered":"2023-12-16T21:13:37.486745+0000","last_clean":"2023-12-16T21:13:37.486745+0000","last_became_active":"2023-12-16T21:13:36.512196+0000","last_became_peered":"2023-12-16T21:13:36.512196+0000","last_unstale":"2023-12-16T21:13:37.486745+0000","last_undegraded":"2023-12-16T21:13:37.486745+0000","last_fullsized":"2023-12-16T21:13:37.486745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:04.676737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580134+0000","last_change":"2023-12-16T21:13:35.511999+0000","last_active":"2023-12-16T21:13:37.580134+0000","last_peered":"2023-12-16T21:13:37.580134+0000","last_clean":"2023-12-16T21:13:37.580134+0000","last_became_active":"2023-12-16T21:13:35.511760+0000","last_became_peered":"2023-12-16T21:13:35.511760+0000","last_unstale":"2023-12-16T21:13:37.580134+0000","last_undegraded":"2023-12-16T21:13:37.580134+0000","last_fullsized":"2023-12-16T21:13:37.580134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:33:49.269497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.487061+0000","last_change":"2023-12-16T21:13:36.512673+0000","last_active":"2023-12-16T21:13:37.487061+0000","last_peered":"2023-12-16T21:13:37.487061+0000","last_clean":"2023-12-16T21:13:37.487061+0000","last_became_active":"2023-12-16T21:13:36.512185+0000","last_became_peered":"2023-12-16T21:13:36.512185+0000","last_unstale":"2023-12-16T21:13:37.487061+0000","last_undegraded":"2023-12-16T21:13:37.487061+0000","last_fullsized":"2023-12-16T21:13:37.487061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:50:05.145896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.580255+0000","last_change":"2023-12-16T21:13:35.511982+0000","last_active":"2023-12-16T21:13:37.580255+0000","last_peered":"2023-12-16T21:13:37.580255+0000","last_clean":"2023-12-16T21:13:37.580255+0000","last_became_active":"2023-12-16T21:13:35.511359+0000","last_became_peered":"2023-12-16T21:13:35.511359+0000","last_unstale":"2023-12-16T21:13:37.580255+0000","last_undegraded":"2023-12-16T21:13:37.580255+0000","last_fullsized":"2023-12-16T21:13:37.580255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:06:19.358431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:38.004907+0000","last_change":"2023-12-16T21:13:36.505890+0000","last_active":"2023-12-16T21:13:38.004907+0000","last_peered":"2023-12-16T21:13:38.004907+0000","last_clean":"2023-12-16T21:13:38.004907+0000","last_became_active":"2023-12-16T21:13:36.505414+0000","last_became_peered":"2023-12-16T21:13:36.505414+0000","last_unstale":"2023-12-16T21:13:38.004907+0000","last_undegraded":"2023-12-16T21:13:38.004907+0000","last_fullsized":"2023-12-16T21:13:38.004907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:12:22.668786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485644+0000","last_change":"2023-12-16T21:13:35.511468+0000","last_active":"2023-12-16T21:13:37.485644+0000","last_peered":"2023-12-16T21:13:37.485644+0000","last_clean":"2023-12-16T21:13:37.485644+0000","last_became_active":"2023-12-16T21:13:35.511092+0000","last_became_peered":"2023-12-16T21:13:35.511092+0000","last_unstale":"2023-12-16T21:13:37.485644+0000","last_undegraded":"2023-12-16T21:13:37.485644+0000","last_fullsized":"2023-12-16T21:13:37.485644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:32:32.837273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.485618+0000","last_change":"2023-12-16T21:13:36.494416+0000","last_active":"2023-12-16T21:13:37.485618+0000","last_peered":"2023-12-16T21:13:37.485618+0000","last_clean":"2023-12-16T21:13:37.485618+0000","last_became_active":"2023-12-16T21:13:36.494121+0000","last_became_peered":"2023-12-16T21:13:36.494121+0000","last_unstale":"2023-12-16T21:13:37.485618+0000","last_undegraded":"2023-12-16T21:13:37.485618+0000","last_fullsized":"2023-12-16T21:13:37.485618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:35.487118+0000","last_clean_scrub_stamp":"2023-12-16T21:13:35.487118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:45:18.716156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-16T21:13:37.968502+0000","last_change":"2023-12-16T21:13:35.510139+0000","last_active":"2023-12-16T21:13:37.968502+0000","last_peered":"2023-12-16T21:13:37.968502+0000","last_clean":"2023-12-16T21:13:37.968502+0000","last_became_active":"2023-12-16T21:13:35.509804+0000","last_became_peered":"2023-12-16T21:13:35.509804+0000","last_unstale":"2023-12-16T21:13:37.968502+0000","last_undegraded":"2023-12-16T21:13:37.968502+0000","last_fullsized":"2023-12-16T21:13:37.968502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:34.483204+0000","last_clean_scrub_stamp":"2023-12-16T21:13:34.483204+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:56:59.813907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:13:48.730 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-16T21:13:48.730 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-16T21:13:48.730 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-16T21:13:48.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-16T21:13:48.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.872+0000 7f779e19e640 1 Processor -- start 2023-12-16T21:13:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.872+0000 7f779e19e640 1 -- start start 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779e19e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f779815d540 0x7f77980ce970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779e19e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f77980d15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779e19e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77980cdca0 con 0x7f779815d160 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779e19e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77980d1390 con 0x7f779815d540 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779e19e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77980cded0 con 0x7f77980d0bb0 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f779815d540 0x7f77980ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f779815d540 0x7f77980ce970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36506/0 (socket says 172.21.15.46:36506) 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 -- 172.21.15.46:0/3087145765 learned_addr learned my addr 172.21.15.46:0/3087145765 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779d99d640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f77980d15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 -- 172.21.15.46:0/3087145765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 msgr2=0x7f77980d15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f77980d15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 -- 172.21.15.46:0/3087145765 >> v1:172.21.15.121:6789/0 conn(0x7f779815d160 legacy=0x7f77980d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779c99b640 1 -- 172.21.15.46:0/3087145765 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77980d10b0 con 0x7f779815d540 2023-12-16T21:13:48.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.873+0000 7f779d99d640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f77980d15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.874+0000 7f779c99b640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f77980ce970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7788009870 tx=0x7f778802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8a36ea100ef1b9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.874+0000 7f778ffff640 1 -- 172.21.15.46:0/3087145765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788031640 con 0x7f779815d540 2023-12-16T21:13:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.874+0000 7f778ffff640 1 -- 172.21.15.46:0/3087145765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77880317a0 con 0x7f779815d540 2023-12-16T21:13:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.874+0000 7f778ffff640 1 -- 172.21.15.46:0/3087145765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788031b30 con 0x7f779815d540 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 -- 172.21.15.46:0/3087145765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 msgr2=0x7f77980ce970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f77980ce970 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7788009870 tx=0x7f778802b8f0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 -- 172.21.15.46:0/3087145765 shutdown_connections 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f77980d15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 --2- 172.21.15.46:0/3087145765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f77980ce970 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 -- 172.21.15.46:0/3087145765 >> 172.21.15.46:0/3087145765 conn(0x7f77980d2fe0 msgr2=0x7f77980d33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 -- 172.21.15.46:0/3087145765 shutdown_connections 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.875+0000 7f779e19e640 1 -- 172.21.15.46:0/3087145765 wait complete. 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.876+0000 7f779e19e640 1 Processor -- start 2023-12-16T21:13:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.876+0000 7f779e19e640 1 -- start start 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779e19e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f7798154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779e19e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f779815d160 0x7f7798159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779e19e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f779815a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779e19e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7798155360 con 0x7f779815d540 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779e19e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77980ca150 con 0x7f779815d160 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779e19e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77980ca430 con 0x7f77980d0bb0 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f779815a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f779815a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36508/0 (socket says 172.21.15.46:36508) 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 -- 172.21.15.46:0/2489022384 learned_addr learned my addr 172.21.15.46:0/2489022384 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779c99b640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f779815d160 0x7f7798159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d19c640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f7798154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 -- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 msgr2=0x7f7798154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f7798154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 -- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f779815d160 msgr2=0x7f7798159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f779815d160 0x7f7798159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.877+0000 7f779d99d640 1 -- 172.21.15.46:0/2489022384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77880094d0 con 0x7f779815d540 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.878+0000 7f779d99d640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f779815a210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f779000cef0 tx=0x7f779000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.878+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779000d840 con 0x7f779815d540 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.878+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779000d9c0 con 0x7f779815d540 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.878+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77980ca710 con 0x7f779815d540 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.878+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779000dd70 con 0x7f779815d540 2023-12-16T21:13:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.878+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77980c7940 con 0x7f779815d540 2023-12-16T21:13:48.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.879+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77980cac30 con 0x7f779815d540 2023-12-16T21:13:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.879+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f779001e3b0 con 0x7f779815d540 2023-12-16T21:13:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.880+0000 7f778dffb640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f777c06ae40 0x7f777c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.880+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7790089fc0 con 0x7f779815d540 2023-12-16T21:13:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.880+0000 7f779d19c640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f777c06ae40 0x7f777c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.880+0000 7f779d19c640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f777c06ae40 0x7f777c06d300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7794004f60 tx=0x7f7794007e30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:48.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:48.884+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f77900bcea0 con 0x7f779815d540 2023-12-16T21:13:49.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.096+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7798069320 con 0x7f779815d540 2023-12-16T21:13:49.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.097+0000 7f778dffb640 1 -- 172.21.15.46:0/2489022384 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f779001a040 con 0x7f779815d540 2023-12-16T21:13:49.096 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:49.097 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T21:13:49.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f777c06ae40 msgr2=0x7f777c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f777c06ae40 0x7f777c06d300 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7794004f60 tx=0x7f7794007e30 comp rx=0 tx=0).stop 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 msgr2=0x7f779815a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f779815a210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f779000cef0 tx=0x7f779000a710 comp rx=0 tx=0).stop 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 shutdown_connections 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f777c06ae40 0x7f777c06d300 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f779815d540 0x7f779815a210 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f779815d160 0x7f7798159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 --2- 172.21.15.46:0/2489022384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77980d0bb0 0x7f7798154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 >> 172.21.15.46:0/2489022384 conn(0x7f77980d2fe0 msgr2=0x7f77980b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 shutdown_connections 2023-12-16T21:13:49.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.099+0000 7f779e19e640 1 -- 172.21.15.46:0/2489022384 wait complete. 2023-12-16T21:13:49.109 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-16T21:13:49.109 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:49.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.217+0000 7f7129d0a640 1 Processor -- start 2023-12-16T21:13:49.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.217+0000 7f7129d0a640 1 -- start start 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.218+0000 7f7129d0a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f71240cf320 0x7f71240d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.218+0000 7f7129d0a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240b3f10 con 0x7f712415f760 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.218+0000 7f7129d0a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240d0a50 con 0x7f712415b130 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.218+0000 7f7129d0a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f712415fd20 con 0x7f71240cf320 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.218+0000 7f7123fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f712415f760 0x7f71240cd310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53594/0 (socket says 172.21.15.46:53594) 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.218+0000 7f7123fff640 1 -- 172.21.15.46:0/2725437103 learned_addr learned my addr 172.21.15.46:0/2725437103 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.219+0000 7f7122ffd640 1 --2- 172.21.15.46:0/2725437103 >> v2:172.21.15.121:3300/0 conn(0x7f71240cf320 0x7f71240d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.219+0000 7f71227fc640 1 -- 172.21.15.46:0/2725437103 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741307137 0 0) 0x7f71240b3f10 con 0x7f712415f760 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.219+0000 7f71227fc640 1 -- 172.21.15.46:0/2725437103 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7110003520 con 0x7f712415f760 2023-12-16T21:13:49.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.219+0000 7f7122ffd640 1 -- 172.21.15.46:0/2725437103 >> v1:172.21.15.121:6789/0 conn(0x7f712415b130 legacy=0x7f71240cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.219+0000 7f7122ffd640 1 -- 172.21.15.46:0/2725437103 >> v1:172.21.15.46:6789/0 conn(0x7f712415f760 legacy=0x7f71240cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.219+0000 7f7122ffd640 1 -- 172.21.15.46:0/2725437103 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240d0d30 con 0x7f71240cf320 2023-12-16T21:13:49.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f7122ffd640 1 --2- 172.21.15.46:0/2725437103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f71240d01f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097a0 tx=0x7f710c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d411c251e3958bb9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f71227fc640 1 -- 172.21.15.46:0/2725437103 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c030610 con 0x7f71240cf320 2023-12-16T21:13:49.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f71227fc640 1 -- 172.21.15.46:0/2725437103 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f710c030770 con 0x7f71240cf320 2023-12-16T21:13:49.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f71227fc640 1 -- 172.21.15.46:0/2725437103 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f710c030ae0 con 0x7f71240cf320 2023-12-16T21:13:49.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f7129d0a640 1 -- 172.21.15.46:0/2725437103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 msgr2=0x7f71240d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f7129d0a640 1 --2- 172.21.15.46:0/2725437103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f71240d01f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097a0 tx=0x7f710c02b960 comp rx=0 tx=0).stop 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.220+0000 7f7129d0a640 1 -- 172.21.15.46:0/2725437103 shutdown_connections 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.221+0000 7f7129d0a640 1 --2- 172.21.15.46:0/2725437103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f71240d01f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.221+0000 7f7129d0a640 1 -- 172.21.15.46:0/2725437103 >> 172.21.15.46:0/2725437103 conn(0x7f71240b2c10 msgr2=0x7f71240b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.221+0000 7f7129d0a640 1 -- 172.21.15.46:0/2725437103 shutdown_connections 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.221+0000 7f7129d0a640 1 -- 172.21.15.46:0/2725437103 wait complete. 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.222+0000 7f7129d0a640 1 Processor -- start 2023-12-16T21:13:49.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.222+0000 7f7129d0a640 1 -- start start 2023-12-16T21:13:49.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f7124154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712415b130 0x7f7124156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 0x7f71241f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712406c300 con 0x7f712415f760 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712406c5e0 con 0x7f71240cf320 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712406c8c0 con 0x7f712415b130 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 0x7f71241f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 0x7f71241f2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36518/0 (socket says 172.21.15.46:36518) 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 -- 172.21.15.46:0/3819427963 learned_addr learned my addr 172.21.15.46:0/3819427963 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7122ffd640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712415b130 0x7f7124156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f71237fe640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f7124154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 -- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712415b130 msgr2=0x7f7124156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712415b130 0x7f7124156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 -- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 msgr2=0x7f7124154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f7124154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 -- 172.21.15.46:0/3819427963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c009400 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7123fff640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 0x7f71241f2e60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f711c00caa0 tx=0x7f711c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c00ac00 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712406cba0 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711c00ad60 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c012620 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.223+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712406bf40 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.224+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7124067fc0 con 0x7f712415f760 2023-12-16T21:13:49.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.225+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f711c007070 con 0x7f712415f760 2023-12-16T21:13:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.225+0000 7f711bfff640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f710406ae40 0x7f710406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.225+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f711c089680 con 0x7f712415f760 2023-12-16T21:13:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.225+0000 7f71237fe640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f710406ae40 0x7f710406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.226+0000 7f71237fe640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f710406ae40 0x7f710406d300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71240d1ff0 tx=0x7f7114009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.229+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f711c0546a0 con 0x7f712415f760 2023-12-16T21:13:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.405+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f712406d0c0 con 0x7f712415f760 2023-12-16T21:13:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.407+0000 7f711bfff640 1 -- 172.21.15.46:0/3819427963 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f712406d0c0 con 0x7f712415f760 2023-12-16T21:13:49.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:49.407 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f710406ae40 msgr2=0x7f710406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f710406ae40 0x7f710406d300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f71240d1ff0 tx=0x7f7114009040 comp rx=0 tx=0).stop 2023-12-16T21:13:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 msgr2=0x7f71241f2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 0x7f71241f2e60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f711c00caa0 tx=0x7f711c00cf60 comp rx=0 tx=0).stop 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 shutdown_connections 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f710406ae40 0x7f710406d300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f712415f760 0x7f71241f2e60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f712415b130 0x7f7124156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 --2- 172.21.15.46:0/3819427963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71240cf320 0x7f7124154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.408+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 >> 172.21.15.46:0/3819427963 conn(0x7f71240b2c10 msgr2=0x7f71240b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:49.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.409+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 shutdown_connections 2023-12-16T21:13:49.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.409+0000 7f7129d0a640 1 -- 172.21.15.46:0/3819427963 wait complete. 2023-12-16T21:13:49.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:13:49.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:13:49.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.528+0000 7fa60cc32640 1 Processor -- start 2023-12-16T21:13:49.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60cc32640 1 -- start start 2023-12-16T21:13:49.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60cc32640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa6080d2910 0x7fa6080d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60cc32640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d3730 0x7fa6080b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60cc32640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080b2b10 con 0x7fa60815d330 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60cc32640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080d1590 con 0x7fa6080d2910 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60cc32640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080d1870 con 0x7fa6080d3730 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa60815d330 0x7fa6080b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53602/0 (socket says 172.21.15.46:53602) 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.529+0000 7fa60659c640 1 -- 172.21.15.46:0/588705811 learned_addr learned my addr 172.21.15.46:0/588705811 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa606d9d640 1 --2- 172.21.15.46:0/588705811 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d3730 0x7fa6080b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa5fffff640 1 --2- 172.21.15.46:0/588705811 >> v2:172.21.15.121:3300/0 conn(0x7fa6080d2910 0x7fa6080d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa605d9b640 1 -- 172.21.15.46:0/588705811 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817673603 0 0) 0x7fa6080b2b10 con 0x7fa60815d330 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa605d9b640 1 -- 172.21.15.46:0/588705811 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f4003520 con 0x7fa60815d330 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa606d9d640 1 -- 172.21.15.46:0/588705811 >> v2:172.21.15.121:3300/0 conn(0x7fa6080d2910 msgr2=0x7fa6080d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa606d9d640 1 --2- 172.21.15.46:0/588705811 >> v2:172.21.15.121:3300/0 conn(0x7fa6080d2910 0x7fa6080d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa606d9d640 1 -- 172.21.15.46:0/588705811 >> v1:172.21.15.46:6789/0 conn(0x7fa60815d330 legacy=0x7fa6080b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.530+0000 7fa606d9d640 1 -- 172.21.15.46:0/588705811 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6080d3b90 con 0x7fa6080d3730 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa606d9d640 1 --2- 172.21.15.46:0/588705811 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d3730 0x7fa6080b6390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa600009d90 tx=0x7fa60002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1f76071a7df4cc2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa605d9b640 1 -- 172.21.15.46:0/588705811 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60002cbc0 con 0x7fa6080d3730 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa605d9b640 1 -- 172.21.15.46:0/588705811 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60002cd20 con 0x7fa6080d3730 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa605d9b640 1 -- 172.21.15.46:0/588705811 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa600034640 con 0x7fa6080d3730 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa60cc32640 1 -- 172.21.15.46:0/588705811 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d3730 msgr2=0x7fa6080b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa60cc32640 1 --2- 172.21.15.46:0/588705811 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d3730 0x7fa6080b6390 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa600009d90 tx=0x7fa60002be10 comp rx=0 tx=0).stop 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa60cc32640 1 -- 172.21.15.46:0/588705811 shutdown_connections 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa60cc32640 1 --2- 172.21.15.46:0/588705811 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d3730 0x7fa6080b6390 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa60cc32640 1 --2- 172.21.15.46:0/588705811 >> v2:172.21.15.121:3300/0 conn(0x7fa6080d2910 0x7fa6080d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.531+0000 7fa60cc32640 1 -- 172.21.15.46:0/588705811 >> 172.21.15.46:0/588705811 conn(0x7fa608072020 msgr2=0x7fa608072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.532+0000 7fa60cc32640 1 -- 172.21.15.46:0/588705811 shutdown_connections 2023-12-16T21:13:49.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.532+0000 7fa60cc32640 1 -- 172.21.15.46:0/588705811 wait complete. 2023-12-16T21:13:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.533+0000 7fa60cc32640 1 Processor -- start 2023-12-16T21:13:49.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.533+0000 7fa60cc32640 1 -- start start 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d2910 0x7fa608160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 0x7fa6080ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa60815d330 0x7fa60815f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608145360 con 0x7fa6080d3730 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081609b0 con 0x7fa60815d330 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081f68c0 con 0x7fa6080d2910 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 0x7fa6080ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 0x7fa6080ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36522/0 (socket says 172.21.15.46:36522) 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 -- 172.21.15.46:0/2959999549 learned_addr learned my addr 172.21.15.46:0/2959999549 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60659c640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d2910 0x7fa608160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa606d9d640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa60815d330 0x7fa60815f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 -- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d2910 msgr2=0x7fa608160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d2910 0x7fa608160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 -- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa60815d330 msgr2=0x7fa60815f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa60815d330 0x7fa60815f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 -- 172.21.15.46:0/2959999549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa600009a70 con 0x7fa6080d3730 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5fffff640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 0x7fa6080ccb60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e8a0 tx=0x7fa5f000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00094f0 con 0x7fa6080d3730 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f000c940 con 0x7fa6080d3730 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6081f6ba0 con 0x7fa6080d3730 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.534+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f000ccf0 con 0x7fa6080d3730 2023-12-16T21:13:49.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.535+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6081f7160 con 0x7fa6080d3730 2023-12-16T21:13:49.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.536+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6080d1df0 con 0x7fa6080d3730 2023-12-16T21:13:49.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.536+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa5f0013470 con 0x7fa6080d3730 2023-12-16T21:13:49.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.536+0000 7fa5feffd640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa5e806ae40 0x7fa5e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.537+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5f0089580 con 0x7fa6080d3730 2023-12-16T21:13:49.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.537+0000 7fa60659c640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa5e806ae40 0x7fa5e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.537+0000 7fa60659c640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa5e806ae40 0x7fa5e806d300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8005880 tx=0x7fa5f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.541+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa5f0054770 con 0x7fa6080d3730 2023-12-16T21:13:49.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.717+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa608069400 con 0x7fa6080d3730 2023-12-16T21:13:49.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.718+0000 7fa5feffd640 1 -- 172.21.15.46:0/2959999549 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5f0057d90 con 0x7fa6080d3730 2023-12-16T21:13:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa5e806ae40 msgr2=0x7fa5e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa5e806ae40 0x7fa5e806d300 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8005880 tx=0x7fa5f8009040 comp rx=0 tx=0).stop 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 msgr2=0x7fa6080ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 0x7fa6080ccb60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000e8a0 tx=0x7fa5f000ed60 comp rx=0 tx=0).stop 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 shutdown_connections 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa5e806ae40 0x7fa5e806d300 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6080d3730 0x7fa6080ccb60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.719+0000 7fa60cc32640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa60815d330 0x7fa60815f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.720+0000 7fa60cc32640 1 --2- 172.21.15.46:0/2959999549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6080d2910 0x7fa608160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.720+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 >> 172.21.15.46:0/2959999549 conn(0x7fa608072020 msgr2=0x7fa60815c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:49.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.720+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 shutdown_connections 2023-12-16T21:13:49.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.720+0000 7fa60cc32640 1 -- 172.21.15.46:0/2959999549 wait complete. 2023-12-16T21:13:49.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:13:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.840+0000 7f6590451640 1 Processor -- start 2023-12-16T21:13:49.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.840+0000 7f6590451640 1 -- start start 2023-12-16T21:13:49.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f6590451640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f65880b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f6590451640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f658815e730 0x7f65880b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f6590451640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f658815a4e0 0x7f658815a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f6590451640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658815b520 con 0x7f658815a4e0 2023-12-16T21:13:49.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f6590451640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6588143870 con 0x7f658815e730 2023-12-16T21:13:49.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f6590451640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658815e2a0 con 0x7f6588143490 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f658815a4e0 0x7f658815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f658815a4e0 0x7f658815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36528/0 (socket says 172.21.15.46:36528) 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 -- 172.21.15.46:0/1949056382 learned_addr learned my addr 172.21.15.46:0/1949056382 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e1ed640 1 --2- 172.21.15.46:0/1949056382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f65880b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658d9ec640 1 --2- 172.21.15.46:0/1949056382 >> v2:172.21.15.121:3300/0 conn(0x7f658815e730 0x7f65880b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 -- 172.21.15.46:0/1949056382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 msgr2=0x7f65880b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 --2- 172.21.15.46:0/1949056382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f65880b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 -- 172.21.15.46:0/1949056382 >> v2:172.21.15.121:3300/0 conn(0x7f658815e730 msgr2=0x7f65880b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 --2- 172.21.15.46:0/1949056382 >> v2:172.21.15.121:3300/0 conn(0x7f658815e730 0x7f65880b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658e9ee640 1 -- 172.21.15.46:0/1949056382 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658815b240 con 0x7f658815a4e0 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.841+0000 7f658d9ec640 1 --2- 172.21.15.46:0/1949056382 >> v2:172.21.15.121:3300/0 conn(0x7f658815e730 0x7f65880b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:49.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.842+0000 7f658e9ee640 1 --2- 172.21.15.46:0/1949056382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f658815a970 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f658806e680 tx=0x7f658002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a6612a2dd14229 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.842+0000 7f658d1eb640 1 -- 172.21.15.46:0/1949056382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658002ea80 con 0x7f658815a4e0 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.842+0000 7f658d1eb640 1 -- 172.21.15.46:0/1949056382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658002ebe0 con 0x7f658815a4e0 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.842+0000 7f658d1eb640 1 -- 172.21.15.46:0/1949056382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580009600 con 0x7f658815a4e0 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 -- 172.21.15.46:0/1949056382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 msgr2=0x7f658815a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 --2- 172.21.15.46:0/1949056382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f658815a970 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f658806e680 tx=0x7f658002e0e0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 -- 172.21.15.46:0/1949056382 shutdown_connections 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 --2- 172.21.15.46:0/1949056382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f658815a970 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 --2- 172.21.15.46:0/1949056382 >> v2:172.21.15.121:3300/0 conn(0x7f658815e730 0x7f65880b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 --2- 172.21.15.46:0/1949056382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f65880b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 -- 172.21.15.46:0/1949056382 >> 172.21.15.46:0/1949056382 conn(0x7f658806a4d0 msgr2=0x7f658806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 -- 172.21.15.46:0/1949056382 shutdown_connections 2023-12-16T21:13:49.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.843+0000 7f6590451640 1 -- 172.21.15.46:0/1949056382 wait complete. 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.844+0000 7f6590451640 1 Processor -- start 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.844+0000 7f6590451640 1 -- start start 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f658813ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f65880add30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f658815e730 0x7f65880b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880aca40 con 0x7f658815a4e0 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658813f370 con 0x7f658815e730 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65880acc90 con 0x7f6588143490 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f65880add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f65880add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36536/0 (socket says 172.21.15.46:36536) 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 -- 172.21.15.46:0/4169001362 learned_addr learned my addr 172.21.15.46:0/4169001362 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 -- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 msgr2=0x7f658813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658e1ed640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f658813ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658e9ee640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f658815e730 0x7f65880b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f658813ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 -- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f658815e730 msgr2=0x7f65880b0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f658815e730 0x7f65880b0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 -- 172.21.15.46:0/4169001362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658000bd40 con 0x7f658815a4e0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658e1ed640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f658813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658e9ee640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f658815e730 0x7f65880b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f658d9ec640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f65880add30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f657800c820 tx=0x7f657800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578007a30 con 0x7f658815a4e0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578007b90 con 0x7f658815a4e0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65880acf70 con 0x7f658815a4e0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657800ce10 con 0x7f658815a4e0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.845+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65880ad530 con 0x7f658815a4e0 2023-12-16T21:13:49.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.847+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6588069400 con 0x7f658815a4e0 2023-12-16T21:13:49.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.847+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f65780113f0 con 0x7f658815a4e0 2023-12-16T21:13:49.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.847+0000 7f657effd640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f656c06ae40 0x7f656c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:49.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.847+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65780896c0 con 0x7f658815a4e0 2023-12-16T21:13:49.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.847+0000 7f658e1ed640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f656c06ae40 0x7f656c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:49.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.848+0000 7f658e1ed640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f656c06ae40 0x7f656c06d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6584006fd0 tx=0x7f6584008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:49.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:49.851+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6578054770 con 0x7f658815a4e0 2023-12-16T21:13:50.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.032+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f65880b2820 con 0x7f658815a4e0 2023-12-16T21:13:50.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.032+0000 7f657effd640 1 -- 172.21.15.46:0/4169001362 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6578057d90 con 0x7f658815a4e0 2023-12-16T21:13:50.032 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:13:50.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f656c06ae40 msgr2=0x7f656c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f656c06ae40 0x7f656c06d300 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6584006fd0 tx=0x7f6584008040 comp rx=0 tx=0).stop 2023-12-16T21:13:50.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 msgr2=0x7f65880add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f65880add30 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f657800c820 tx=0x7f657800cce0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 shutdown_connections 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f656c06ae40 0x7f656c06d300 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f658815a4e0 0x7f65880add30 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f658815e730 0x7f65880b0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 --2- 172.21.15.46:0/4169001362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6588143490 0x7f658813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 >> 172.21.15.46:0/4169001362 conn(0x7f658806a4d0 msgr2=0x7f658815ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 shutdown_connections 2023-12-16T21:13:50.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.034+0000 7f6590451640 1 -- 172.21.15.46:0/4169001362 wait complete. 2023-12-16T21:13:50.045 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:13:50.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.154+0000 7f97cf8d3640 1 Processor -- start 2023-12-16T21:13:50.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.154+0000 7f97cf8d3640 1 -- start start 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cf8d3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f97c8160fb0 0x7f97c8161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cf8d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c8161d20 con 0x7f97c80d0ed0 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cf8d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c815e760 con 0x7f97c806ea50 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cf8d3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97c815d9a0 con 0x7f97c8160fb0 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cde70640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97c80d0ed0 0x7f97c80d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:53606/0 (socket says 172.21.15.46:53606) 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cde70640 1 -- 172.21.15.46:0/2740475921 learned_addr learned my addr 172.21.15.46:0/2740475921 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97cce6e640 1 --2- 172.21.15.46:0/2740475921 >> v2:172.21.15.121:3300/0 conn(0x7f97c8160fb0 0x7f97c8161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870705122 0 0) 0x7f97c8161d20 con 0x7f97c80d0ed0 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.155+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97b4003520 con 0x7f97c80d0ed0 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2133766030 0 0) 0x7f97b4003520 con 0x7f97c80d0ed0 2023-12-16T21:13:50.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 >> v2:172.21.15.121:3300/0 conn(0x7f97c8160fb0 msgr2=0x7f97c8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 --2- 172.21.15.46:0/2740475921 >> v2:172.21.15.121:3300/0 conn(0x7f97c8160fb0 0x7f97c8161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 >> v1:172.21.15.121:6789/0 conn(0x7f97c806ea50 legacy=0x7f97c806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c815dc80 con 0x7f97c80d0ed0 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f97c00019f0 con 0x7f97c80d0ed0 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 >> v1:172.21.15.46:6789/0 conn(0x7f97c80d0ed0 legacy=0x7f97c80d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b40047c0 0x7f97b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b4005180 0x7f97b400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c815d9a0 con 0x7f97b4003970 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b400c120 con 0x7f97b4005180 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b400c2e0 con 0x7f97b40047c0 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cce6e640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b40047c0 0x7f97b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cde70640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b4005180 0x7f97b400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cd66f640 1 -- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b40047c0 msgr2=0x7f97b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b40047c0 0x7f97b4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cd66f640 1 -- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b4005180 msgr2=0x7f97b400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b4005180 0x7f97b400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cd66f640 1 -- 172.21.15.46:0/2740475921 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4003520 con 0x7f97b4003970 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.156+0000 7f97cde70640 1 -- 172.21.15.46:0/2740475921 reap_dead start 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97b4003d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f97c4001750 tx=0x7f97b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f12a376250efa0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c40052d0 con 0x7f97b4003970 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c402e050 con 0x7f97b4003970 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97bffff640 1 -- 172.21.15.46:0/2740475921 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4003830 con 0x7f97b4003970 2023-12-16T21:13:50.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2740475921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 msgr2=0x7f97b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97b4003d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f97c4001750 tx=0x7f97b40057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2740475921 shutdown_connections 2023-12-16T21:13:50.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2740475921 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97b4003d90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.157+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2740475921 >> 172.21.15.46:0/2740475921 conn(0x7f97c806a6b0 msgr2=0x7f97c806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:50.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.158+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2740475921 shutdown_connections 2023-12-16T21:13:50.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.158+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2740475921 wait complete. 2023-12-16T21:13:50.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.159+0000 7f97cf8d3640 1 Processor -- start 2023-12-16T21:13:50.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.159+0000 7f97cf8d3640 1 -- start start 2023-12-16T21:13:50.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.159+0000 7f97cf8d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97c81590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cf8d3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97c8157330 0x7f97c80ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cf8d3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97c8156b70 0x7f97c806c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cf8d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c81578d0 con 0x7f97b4003970 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cf8d3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c8157bb0 con 0x7f97c8157330 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cf8d3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c8157e90 con 0x7f97c8156b70 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97c81590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97c81590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36550/0 (socket says 172.21.15.46:36550) 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 -- 172.21.15.46:0/2206407142 learned_addr learned my addr 172.21.15.46:0/2206407142 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cde70640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97c8156b70 0x7f97c806c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cce6e640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97c8157330 0x7f97c80ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 -- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97c8156b70 msgr2=0x7f97c806c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97c8156b70 0x7f97c806c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 -- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97c8157330 msgr2=0x7f97c80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97c8157330 0x7f97c80ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 -- 172.21.15.46:0/2206407142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8158170 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cce6e640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97c8157330 0x7f97c80ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cd66f640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97c81590d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97c402bc00 tx=0x7f97c4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4007a30 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c4007640 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c80ce5d0 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.160+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4006c00 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.161+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c8156ff0 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.161+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c8069400 con 0x7f97b4003970 2023-12-16T21:13:50.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.162+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f97c40035f0 con 0x7f97b4003970 2023-12-16T21:13:50.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.162+0000 7f97bdffb640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f97ac06ae40 0x7f97ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.162+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97c40ab0d0 con 0x7f97b4003970 2023-12-16T21:13:50.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.162+0000 7f97cce6e640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f97ac06ae40 0x7f97ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.163+0000 7f97cce6e640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f97ac06ae40 0x7f97ac06d300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97c81f6310 tx=0x7f97b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.166+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f97c4079ca0 con 0x7f97b4003970 2023-12-16T21:13:50.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.343+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97c8070d30 con 0x7f97b4003970 2023-12-16T21:13:50.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.344+0000 7f97bdffb640 1 -- 172.21.15.46:0/2206407142 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f97c4079830 con 0x7f97b4003970 2023-12-16T21:13:50.343 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:13:50.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f97ac06ae40 msgr2=0x7f97ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f97ac06ae40 0x7f97ac06d300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f97c81f6310 tx=0x7f97b8008040 comp rx=0 tx=0).stop 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 msgr2=0x7f97c81590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97c81590d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f97c402bc00 tx=0x7f97c4001120 comp rx=0 tx=0).stop 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 shutdown_connections 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f97ac06ae40 0x7f97ac06d300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97c8156b70 0x7f97c806c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97c8157330 0x7f97c80ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 --2- 172.21.15.46:0/2206407142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97b4003970 0x7f97c81590d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 >> 172.21.15.46:0/2206407142 conn(0x7f97c806a6b0 msgr2=0x7f97c806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 shutdown_connections 2023-12-16T21:13:50.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.346+0000 7f97cf8d3640 1 -- 172.21.15.46:0/2206407142 wait complete. 2023-12-16T21:13:50.356 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:13:50.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.462+0000 7f9a63ad0640 1 Processor -- start 2023-12-16T21:13:50.462 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.462+0000 7f9a63ad0640 1 -- start start 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a63ad0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 0x7f9a5c1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a63ad0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a5c1409d0 0x7f9a5c06a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a63ad0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a5c1406f0 con 0x7f9a5c1428d0 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a63ad0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5c161e20 con 0x7f9a5c1409d0 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a63ad0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c143960 con 0x7f9a5c140310 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a6186c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 0x7f9a5c1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a6106b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a5c1409d0 0x7f9a5c06a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a6186c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 0x7f9a5c1618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:51506/0 (socket says 172.21.15.46:51506) 2023-12-16T21:13:50.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.463+0000 7f9a6186c640 1 -- 172.21.15.46:0/2860422224 learned_addr learned my addr 172.21.15.46:0/2860422224 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:50.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.464+0000 7f9a6186c640 1 -- 172.21.15.46:0/2860422224 >> v2:172.21.15.121:3300/0 conn(0x7f9a5c1409d0 msgr2=0x7f9a5c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:50.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.464+0000 7f9a6186c640 1 --2- 172.21.15.46:0/2860422224 >> v2:172.21.15.121:3300/0 conn(0x7f9a5c1409d0 0x7f9a5c06a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.464+0000 7f9a6186c640 1 -- 172.21.15.46:0/2860422224 >> v1:172.21.15.121:6789/0 conn(0x7f9a5c1428d0 legacy=0x7f9a5c0714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.464+0000 7f9a6186c640 1 -- 172.21.15.46:0/2860422224 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c142de0 con 0x7f9a5c140310 2023-12-16T21:13:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.464+0000 7f9a6186c640 1 --2- 172.21.15.46:0/2860422224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 0x7f9a5c1618e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009e50 tx=0x7f9a4c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea956aac44ca9c7a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.464+0000 7f9a6086a640 1 -- 172.21.15.46:0/2860422224 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c02ca70 con 0x7f9a5c140310 2023-12-16T21:13:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a6086a640 1 -- 172.21.15.46:0/2860422224 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a4c02cbd0 con 0x7f9a5c140310 2023-12-16T21:13:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a6086a640 1 -- 172.21.15.46:0/2860422224 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c0343f0 con 0x7f9a5c140310 2023-12-16T21:13:50.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 -- 172.21.15.46:0/2860422224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 msgr2=0x7f9a5c1618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/2860422224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 0x7f9a5c1618e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c009e50 tx=0x7f9a4c02bed0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 -- 172.21.15.46:0/2860422224 shutdown_connections 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/2860422224 >> v2:172.21.15.121:3300/0 conn(0x7f9a5c1409d0 0x7f9a5c06a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/2860422224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c140310 0x7f9a5c1618e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 -- 172.21.15.46:0/2860422224 >> 172.21.15.46:0/2860422224 conn(0x7f9a5c0b1e50 msgr2=0x7f9a5c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 -- 172.21.15.46:0/2860422224 shutdown_connections 2023-12-16T21:13:50.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.465+0000 7f9a63ad0640 1 -- 172.21.15.46:0/2860422224 wait complete. 2023-12-16T21:13:50.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.466+0000 7f9a63ad0640 1 Processor -- start 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.466+0000 7f9a63ad0640 1 -- start start 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a63ad0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a5c140310 0x7f9a5c1302b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a63ad0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c1409d0 0x7f9a5c13d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a63ad0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 0x7f9a5c0adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a63ad0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1546e0 con 0x7f9a5c1428d0 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a63ad0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c0b0880 con 0x7f9a5c140310 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a63ad0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c1307f0 con 0x7f9a5c1409d0 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a6206d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 0x7f9a5c0adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a6206d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 0x7f9a5c0adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36552/0 (socket says 172.21.15.46:36552) 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a6206d640 1 -- 172.21.15.46:0/1104990676 learned_addr learned my addr 172.21.15.46:0/1104990676 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a6106b640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c1409d0 0x7f9a5c13d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.467+0000 7f9a6186c640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a5c140310 0x7f9a5c1302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6206d640 1 -- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c1409d0 msgr2=0x7f9a5c13d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6206d640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c1409d0 0x7f9a5c13d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6206d640 1 -- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a5c140310 msgr2=0x7f9a5c1302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6206d640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a5c140310 0x7f9a5c1302b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6206d640 1 -- 172.21.15.46:0/1104990676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4c009b30 con 0x7f9a5c1428d0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6106b640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c1409d0 0x7f9a5c13d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a6206d640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 0x7f9a5c0adaf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c400 tx=0x7f9a5400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a540052d0 con 0x7f9a5c1428d0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a54005430 con 0x7f9a5c1428d0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c12dcd0 con 0x7f9a5c1428d0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a54003810 con 0x7f9a5c1428d0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.468+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c12e290 con 0x7f9a5c1428d0 2023-12-16T21:13:50.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.469+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c069400 con 0x7f9a5c1428d0 2023-12-16T21:13:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.470+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9a54003970 con 0x7f9a5c1428d0 2023-12-16T21:13:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.470+0000 7f9a5a7fc640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4806ae40 0x7f9a4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.470+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a54088b80 con 0x7f9a5c1428d0 2023-12-16T21:13:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.470+0000 7f9a6186c640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4806ae40 0x7f9a4806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.471+0000 7f9a6186c640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4806ae40 0x7f9a4806d300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c035000 tx=0x7f9a4c030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.474+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9a54053c30 con 0x7f9a5c1428d0 2023-12-16T21:13:50.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.689+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a5c06ea60 con 0x7f9a5c1428d0 2023-12-16T21:13:50.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.689+0000 7f9a5a7fc640 1 -- 172.21.15.46:0/1104990676 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f9a54017090 con 0x7f9a5c1428d0 2023-12-16T21:13:50.689 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:50.689 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:13:50.690 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:13:50.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.691+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4806ae40 msgr2=0x7f9a4806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.691+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4806ae40 0x7f9a4806d300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c035000 tx=0x7f9a4c030000 comp rx=0 tx=0).stop 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.691+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 msgr2=0x7f9a5c0adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.691+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 0x7f9a5c0adaf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9a5400c400 tx=0x7f9a5400c8c0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.691+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 shutdown_connections 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.691+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4806ae40 0x7f9a4806d300 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.692+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a5c1428d0 0x7f9a5c0adaf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.692+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a5c1409d0 0x7f9a5c13d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.692+0000 7f9a63ad0640 1 --2- 172.21.15.46:0/1104990676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a5c140310 0x7f9a5c1302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.692+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 >> 172.21.15.46:0/1104990676 conn(0x7f9a5c0b1e50 msgr2=0x7f9a5c06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.692+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 shutdown_connections 2023-12-16T21:13:50.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.692+0000 7f9a63ad0640 1 -- 172.21.15.46:0/1104990676 wait complete. 2023-12-16T21:13:50.702 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:50.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.809+0000 7f1048b1d640 1 Processor -- start 2023-12-16T21:13:50.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.809+0000 7f1048b1d640 1 -- start start 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1048b1d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1048b1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10440ad640 0x7f104406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1048b1d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440adae0 con 0x7f10440aeb80 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1048b1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10440addc0 con 0x7f10440ad640 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1048b1d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10440ae0a0 con 0x7f104415d1d0 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10440ad640 0x7f104406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10440ad640 0x7f104406ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33120/0 (socket says 172.21.15.46:33120) 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 -- 172.21.15.46:0/3059036169 learned_addr learned my addr 172.21.15.46:0/3059036169 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:50.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f104259c640 1 --2- 172.21.15.46:0/3059036169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 -- 172.21.15.46:0/3059036169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 msgr2=0x7f104406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 --2- 172.21.15.46:0/3059036169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104406b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 -- 172.21.15.46:0/3059036169 >> v1:172.21.15.121:6789/0 conn(0x7f10440aeb80 legacy=0x7f1044160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.810+0000 7f1041d9b640 1 -- 172.21.15.46:0/3059036169 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440ae380 con 0x7f10440ad640 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.811+0000 7f1041d9b640 1 --2- 172.21.15.46:0/3059036169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440ad640 0x7f104406ba20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1038009870 tx=0x7f103802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f2cbc3f4748de6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.811+0000 7f104159a640 1 -- 172.21.15.46:0/3059036169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038030840 con 0x7f10440ad640 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.811+0000 7f104159a640 1 -- 172.21.15.46:0/3059036169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10380309a0 con 0x7f10440ad640 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.811+0000 7f104159a640 1 -- 172.21.15.46:0/3059036169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038030d30 con 0x7f10440ad640 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 -- 172.21.15.46:0/3059036169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440ad640 msgr2=0x7f104406ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3059036169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440ad640 0x7f104406ba20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1038009870 tx=0x7f103802b980 comp rx=0 tx=0).stop 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 -- 172.21.15.46:0/3059036169 shutdown_connections 2023-12-16T21:13:50.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3059036169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440ad640 0x7f104406ba20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3059036169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 -- 172.21.15.46:0/3059036169 >> 172.21.15.46:0/3059036169 conn(0x7f1044070e70 msgr2=0x7f1044071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 -- 172.21.15.46:0/3059036169 shutdown_connections 2023-12-16T21:13:50.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.812+0000 7f1048b1d640 1 -- 172.21.15.46:0/3059036169 wait complete. 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.813+0000 7f1048b1d640 1 Processor -- start 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.813+0000 7f1048b1d640 1 -- start start 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1048b1d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440ad640 0x7f1044157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1048b1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 0x7f104415d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1048b1d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104415de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1048b1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10441413d0 con 0x7f10440aeb80 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1048b1d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044157b00 con 0x7f10440ad640 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1048b1d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1044157de0 con 0x7f104415d1d0 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 0x7f104415d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 0x7f104415d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33134/0 (socket says 172.21.15.46:33134) 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 -- 172.21.15.46:0/3232028613 learned_addr learned my addr 172.21.15.46:0/3232028613 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1042d9d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104415de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f104259c640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440ad640 0x7f1044157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 -- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 msgr2=0x7f104415de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104415de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 -- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440ad640 msgr2=0x7f1044157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440ad640 0x7f1044157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f1041d9b640 1 -- 172.21.15.46:0/3232028613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10380094d0 con 0x7f10440aeb80 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.814+0000 7f104259c640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440ad640 0x7f1044157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.815+0000 7f1041d9b640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 0x7f104415d940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1038000c00 tx=0x7f1038004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.815+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038039c90 con 0x7f10440aeb80 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.815+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1038039df0 con 0x7f10440aeb80 2023-12-16T21:13:50.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.815+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10441580c0 con 0x7f10440aeb80 2023-12-16T21:13:50.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.815+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038043640 con 0x7f10440aeb80 2023-12-16T21:13:50.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.815+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1044158680 con 0x7f10440aeb80 2023-12-16T21:13:50.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.816+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1044158870 con 0x7f10440aeb80 2023-12-16T21:13:50.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.816+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f10380437a0 con 0x7f10440aeb80 2023-12-16T21:13:50.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.816+0000 7f1032ffd640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f102406ae40 0x7f102406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:50.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.817+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f103803e070 con 0x7f10440aeb80 2023-12-16T21:13:50.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.817+0000 7f104259c640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f102406ae40 0x7f102406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:50.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.817+0000 7f104259c640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f102406ae40 0x7f102406d300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1034006fd0 tx=0x7f1034008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:50.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.821+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f10380777c0 con 0x7f10440aeb80 2023-12-16T21:13:50.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:50.998+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1044069400 con 0x7f10440aeb80 2023-12-16T21:13:50.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.000+0000 7f1032ffd640 1 -- 172.21.15.46:0/3232028613 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f103807ade0 con 0x7f10440aeb80 2023-12-16T21:13:50.999 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:51.000 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:51.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f102406ae40 msgr2=0x7f102406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f102406ae40 0x7f102406d300 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1034006fd0 tx=0x7f1034008040 comp rx=0 tx=0).stop 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 msgr2=0x7f104415d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 0x7f104415d940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1038000c00 tx=0x7f1038004730 comp rx=0 tx=0).stop 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 shutdown_connections 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f102406ae40 0x7f102406d300 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10440aeb80 0x7f104415d940 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f104415d1d0 0x7f104415de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 --2- 172.21.15.46:0/3232028613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440ad640 0x7f1044157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 >> 172.21.15.46:0/3232028613 conn(0x7f1044070e70 msgr2=0x7f10440b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 shutdown_connections 2023-12-16T21:13:51.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.002+0000 7f1048b1d640 1 -- 172.21.15.46:0/3232028613 wait complete. 2023-12-16T21:13:51.012 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:13:51.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:13:51.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.122+0000 7f34b431d640 1 Processor -- start 2023-12-16T21:13:51.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.122+0000 7f34b431d640 1 -- start start 2023-12-16T21:13:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.123+0000 7f34b431d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 0x7f34ac06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.123+0000 7f34b431d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34ac0d1d50 0x7f34ac06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.123+0000 7f34b431d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac06ce00 con 0x7f34ac15d2e0 2023-12-16T21:13:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.123+0000 7f34b431d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac06bf40 con 0x7f34ac0d1d50 2023-12-16T21:13:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.123+0000 7f34b431d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac06c1f0 con 0x7f34ac15d6c0 2023-12-16T21:13:51.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 0x7f34ac06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 0x7f34ac06b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:41388/0 (socket says 172.21.15.46:41388) 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b28ba640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f34ac0d1d50 0x7f34ac06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 -- 172.21.15.46:0/3398028526 learned_addr learned my addr 172.21.15.46:0/3398028526 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 -- 172.21.15.46:0/3398028526 >> v2:172.21.15.121:3300/0 conn(0x7f34ac0d1d50 msgr2=0x7f34ac06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 --2- 172.21.15.46:0/3398028526 >> v2:172.21.15.121:3300/0 conn(0x7f34ac0d1d50 0x7f34ac06ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 -- 172.21.15.46:0/3398028526 >> v1:172.21.15.121:6789/0 conn(0x7f34ac15d2e0 legacy=0x7f34ac0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.124+0000 7f34b18b8640 1 -- 172.21.15.46:0/3398028526 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac06c4d0 con 0x7f34ac15d6c0 2023-12-16T21:13:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.125+0000 7f34b18b8640 1 --2- 172.21.15.46:0/3398028526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 0x7f34ac06b4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009870 tx=0x7f34a002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e601ef0df9e7021d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.125+0000 7f34b10b7640 1 -- 172.21.15.46:0/3398028526 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0031640 con 0x7f34ac15d6c0 2023-12-16T21:13:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.125+0000 7f34b10b7640 1 -- 172.21.15.46:0/3398028526 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a00317a0 con 0x7f34ac15d6c0 2023-12-16T21:13:51.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.125+0000 7f34b10b7640 1 -- 172.21.15.46:0/3398028526 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0031b30 con 0x7f34ac15d6c0 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.125+0000 7f34b431d640 1 -- 172.21.15.46:0/3398028526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 msgr2=0x7f34ac06b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.125+0000 7f34b431d640 1 --2- 172.21.15.46:0/3398028526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 0x7f34ac06b4c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009870 tx=0x7f34a002b8f0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.126+0000 7f34b431d640 1 -- 172.21.15.46:0/3398028526 shutdown_connections 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.126+0000 7f34b431d640 1 --2- 172.21.15.46:0/3398028526 >> v2:172.21.15.121:3300/0 conn(0x7f34ac0d1d50 0x7f34ac06ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.126+0000 7f34b431d640 1 --2- 172.21.15.46:0/3398028526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac15d6c0 0x7f34ac06b4c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.126+0000 7f34b431d640 1 -- 172.21.15.46:0/3398028526 >> 172.21.15.46:0/3398028526 conn(0x7f34ac09c580 msgr2=0x7f34ac0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.126+0000 7f34b431d640 1 -- 172.21.15.46:0/3398028526 shutdown_connections 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.126+0000 7f34b431d640 1 -- 172.21.15.46:0/3398028526 wait complete. 2023-12-16T21:13:51.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 Processor -- start 2023-12-16T21:13:51.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 -- start start 2023-12-16T21:13:51.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac0d1d50 0x7f34ac0cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 0x7f34ac0ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34ac15d6c0 0x7f34ac150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0ca800 con 0x7f34ac15d2e0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0ca9f0 con 0x7f34ac15d6c0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.127+0000 7f34b431d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac151530 con 0x7f34ac0d1d50 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 0x7f34ac0ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b20b9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac0d1d50 0x7f34ac0cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 0x7f34ac0ca2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33144/0 (socket says 172.21.15.46:33144) 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 -- 172.21.15.46:0/4041701697 learned_addr learned my addr 172.21.15.46:0/4041701697 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b28ba640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34ac15d6c0 0x7f34ac150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 -- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac0d1d50 msgr2=0x7f34ac0cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac0d1d50 0x7f34ac0cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 -- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34ac15d6c0 msgr2=0x7f34ac150ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34ac15d6c0 0x7f34ac150ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 -- 172.21.15.46:0/4041701697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a00094d0 con 0x7f34ac15d2e0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b18b8640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 0x7f34ac0ca2c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f34a002b930 tx=0x7f34a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a003a930 con 0x7f34ac15d2e0 2023-12-16T21:13:51.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a003aa90 con 0x7f34ac15d2e0 2023-12-16T21:13:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac0c8e50 con 0x7f34ac15d2e0 2023-12-16T21:13:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a003ae40 con 0x7f34ac15d2e0 2023-12-16T21:13:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.128+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac0c9410 con 0x7f34ac15d2e0 2023-12-16T21:13:51.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.129+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac067420 con 0x7f34ac15d2e0 2023-12-16T21:13:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.130+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f34a002f050 con 0x7f34ac15d2e0 2023-12-16T21:13:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.130+0000 7f34aaffd640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f349806ae40 0x7f349806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.130+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34a00ab3b0 con 0x7f34ac15d2e0 2023-12-16T21:13:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.130+0000 7f34b20b9640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f349806ae40 0x7f349806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.131+0000 7f34b20b9640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f349806ae40 0x7f349806d300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f34a4001780 tx=0x7f34a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.134+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f34a00766c0 con 0x7f34ac15d2e0 2023-12-16T21:13:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.310+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34ac0d1b30 con 0x7f34ac15d2e0 2023-12-16T21:13:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.310+0000 7f34aaffd640 1 -- 172.21.15.46:0/4041701697 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34a002d040 con 0x7f34ac15d2e0 2023-12-16T21:13:51.310 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f349806ae40 msgr2=0x7f349806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f349806ae40 0x7f349806d300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f34a4001780 tx=0x7f34a4009040 comp rx=0 tx=0).stop 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 msgr2=0x7f34ac0ca2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 0x7f34ac0ca2c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f34a002b930 tx=0x7f34a00046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 shutdown_connections 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f349806ae40 0x7f349806d300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34ac15d2e0 0x7f34ac0ca2c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34ac15d6c0 0x7f34ac150ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 --2- 172.21.15.46:0/4041701697 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34ac0d1d50 0x7f34ac0cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 >> 172.21.15.46:0/4041701697 conn(0x7f34ac09c580 msgr2=0x7f34ac0b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 shutdown_connections 2023-12-16T21:13:51.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.312+0000 7f34b431d640 1 -- 172.21.15.46:0/4041701697 wait complete. 2023-12-16T21:13:51.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:13:51.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.433+0000 7fd6b6dfb640 1 Processor -- start 2023-12-16T21:13:51.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b6dfb640 1 -- start start 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b6dfb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6b00d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b6dfb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b006ce00 con 0x7fd6b006bbd0 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b6dfb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b006c1f0 con 0x7fd6b015d6c0 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b6dfb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b006c4d0 con 0x7fd6b015d2e0 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b5398640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd6b006bbd0 0x7fd6b00afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33390/0 (socket says 172.21.15.46:33390) 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.434+0000 7fd6b5398640 1 -- 172.21.15.46:0/3989566190 learned_addr learned my addr 172.21.15.46:0/3989566190 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6b00d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330087269 0 0) 0x7fd6b006ce00 con 0x7fd6b006bbd0 2023-12-16T21:13:51.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6a0003520 con 0x7fd6b006bbd0 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589221970 0 0) 0x7fd6a0003520 con 0x7fd6b006bbd0 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 msgr2=0x7fd6b00d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6b00d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 >> v1:172.21.15.121:6789/0 conn(0x7fd6b015d6c0 legacy=0x7fd6b006b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b006c7b0 con 0x7fd6b006bbd0 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fd6a80019f0 con 0x7fd6b006bbd0 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6b00d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 >> v1:172.21.15.46:6789/0 conn(0x7fd6b006bbd0 legacy=0x7fd6b00afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6a0004350 0x7fd6a0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6a00050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b006c4d0 con 0x7fd6a0004350 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a000b9d0 con 0x7fd6a00039a0 2023-12-16T21:13:51.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a000bb90 con 0x7fd6b015d2e0 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6affff640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6a0004350 0x7fd6a0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6b5398640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6a00050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6affff640 1 -- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 msgr2=0x7fd6a00050e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6affff640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6a00050e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6affff640 1 -- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6a00039a0 msgr2=0x7fd6a0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6affff640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6affff640 1 -- 172.21.15.46:0/3989566190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a4009400 con 0x7fd6a0004350 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6a00039a0 0x7fd6a0003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.435+0000 7fd6b5398640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6b015d2e0 0x7fd6a00050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.436+0000 7fd6affff640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6a0004350 0x7fd6a0004820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd69c001830 tx=0x7fd6a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8556170a18af1b34 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.436+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c02f050 con 0x7fd6a0004350 2023-12-16T21:13:51.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.436+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c033610 con 0x7fd6a0004350 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.436+0000 7fd6af7fe640 1 -- 172.21.15.46:0/3989566190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c0339c0 con 0x7fd6a0004350 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/3989566190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6a0004350 msgr2=0x7fd6a0004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/3989566190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6a0004350 0x7fd6a0004820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd69c001830 tx=0x7fd6a00057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b5398640 1 -- 172.21.15.46:0/3989566190 reap_dead start 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/3989566190 shutdown_connections 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/3989566190 >> 172.21.15.46:0/3989566190 conn(0x7fd6b009c580 msgr2=0x7fd6b00d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/3989566190 shutdown_connections 2023-12-16T21:13:51.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.437+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/3989566190 wait complete. 2023-12-16T21:13:51.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.438+0000 7fd6b6dfb640 1 Processor -- start 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.438+0000 7fd6b6dfb640 1 -- start start 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b6dfb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6a0004350 0x7fd6b00ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b6dfb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6b00c99a0 0x7fd6b01512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b6dfb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 0x7fd6b0151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b6dfb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b0151d50 con 0x7fd6b00c9e20 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b6dfb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b00b76f0 con 0x7fd6b00c99a0 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b6dfb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015edf0 con 0x7fd6a0004350 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 0x7fd6b0151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 0x7fd6b0151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33162/0 (socket says 172.21.15.46:33162) 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 -- 172.21.15.46:0/218404694 learned_addr learned my addr 172.21.15.46:0/218404694 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6a0004350 0x7fd6b00ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 -- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6a0004350 msgr2=0x7fd6b00ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6a0004350 0x7fd6b00ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 -- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6b00c99a0 msgr2=0x7fd6b01512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6b00c99a0 0x7fd6b01512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 -- 172.21.15.46:0/218404694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b015f930 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6a0004350 0x7fd6b00ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.439+0000 7fd6b5398640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 0x7fd6b0151810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00d2750 tx=0x7fd6a800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.440+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a800eb30 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.440+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a800ec90 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.440+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b015f000 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.440+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8013420 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.440+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b015f5c0 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.441+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b015fac0 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.441+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd6a8013580 con 0x7fd6b00c9e20 2023-12-16T21:13:51.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.441+0000 7fd6ad7fa640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd69406ae70 0x7fd69406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.441+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6a80899a0 con 0x7fd6b00c9e20 2023-12-16T21:13:51.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.442+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd69406ae70 0x7fd69406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.442+0000 7fd6b4b97640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd69406ae70 0x7fd69406d330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009c10 tx=0x7fd6a4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.446+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd6a8054a50 con 0x7fd6b00c9e20 2023-12-16T21:13:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.623+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6b0067420 con 0x7fd6b00c9e20 2023-12-16T21:13:51.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.624+0000 7fd6ad7fa640 1 -- 172.21.15.46:0/218404694 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6a8058070 con 0x7fd6b00c9e20 2023-12-16T21:13:51.624 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd69406ae70 msgr2=0x7fd69406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd69406ae70 0x7fd69406d330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009c10 tx=0x7fd6a4008040 comp rx=0 tx=0).stop 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 msgr2=0x7fd6b0151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 0x7fd6b0151810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd6b00d2750 tx=0x7fd6a800e530 comp rx=0 tx=0).stop 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 shutdown_connections 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd69406ae70 0x7fd69406d330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6b00c9e20 0x7fd6b0151810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6b00c99a0 0x7fd6b01512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 --2- 172.21.15.46:0/218404694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6a0004350 0x7fd6b00ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 >> 172.21.15.46:0/218404694 conn(0x7fd6b009c580 msgr2=0x7fd6b00b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 shutdown_connections 2023-12-16T21:13:51.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.626+0000 7fd6b6dfb640 1 -- 172.21.15.46:0/218404694 wait complete. 2023-12-16T21:13:51.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:13:51.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.747+0000 7f2a2b58e640 1 Processor -- start 2023-12-16T21:13:51.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.747+0000 7f2a2b58e640 1 -- start start 2023-12-16T21:13:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2b58e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a24146460 0x7f2a240b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2b58e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a24146840 0x7f2a2406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2b58e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a2406eab0 0x7f2a2406ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2b58e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a24146fd0 con 0x7f2a24146840 2023-12-16T21:13:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2b58e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2406f440 con 0x7f2a2406eab0 2023-12-16T21:13:51.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2b58e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2406c180 con 0x7f2a24146460 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a28b29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a24146840 0x7f2a2406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a28b29640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a24146840 0x7f2a2406baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33170/0 (socket says 172.21.15.46:33170) 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a28b29640 1 -- 172.21.15.46:0/3628112288 learned_addr learned my addr 172.21.15.46:0/3628112288 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a29b2b640 1 --2- 172.21.15.46:0/3628112288 >> v2:172.21.15.121:3300/0 conn(0x7f2a2406eab0 0x7f2a2406ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a2932a640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a24146460 0x7f2a240b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.748+0000 7f2a28b29640 1 -- 172.21.15.46:0/3628112288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a24146460 msgr2=0x7f2a240b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a28b29640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a24146460 0x7f2a240b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a28b29640 1 -- 172.21.15.46:0/3628112288 >> v2:172.21.15.121:3300/0 conn(0x7f2a2406eab0 msgr2=0x7f2a2406ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:51.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a28b29640 1 --2- 172.21.15.46:0/3628112288 >> v2:172.21.15.121:3300/0 conn(0x7f2a2406eab0 0x7f2a2406ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a28b29640 1 -- 172.21.15.46:0/3628112288 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2406c460 con 0x7f2a24146840 2023-12-16T21:13:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a2932a640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a24146460 0x7f2a240b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a28b29640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a2406baf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2a240b66c0 tx=0x7f2a1802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6400506f116d128 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a23fff640 1 -- 172.21.15.46:0/3628112288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18031860 con 0x7f2a24146840 2023-12-16T21:13:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a23fff640 1 -- 172.21.15.46:0/3628112288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a180319c0 con 0x7f2a24146840 2023-12-16T21:13:51.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.749+0000 7f2a23fff640 1 -- 172.21.15.46:0/3628112288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18031d50 con 0x7f2a24146840 2023-12-16T21:13:51.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 -- 172.21.15.46:0/3628112288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 msgr2=0x7f2a2406baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a2406baf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2a240b66c0 tx=0x7f2a1802b900 comp rx=0 tx=0).stop 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 -- 172.21.15.46:0/3628112288 shutdown_connections 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/3628112288 >> v2:172.21.15.121:3300/0 conn(0x7f2a2406eab0 0x7f2a2406ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a2406baf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/3628112288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a24146460 0x7f2a240b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 -- 172.21.15.46:0/3628112288 >> 172.21.15.46:0/3628112288 conn(0x7f2a2406a6b0 msgr2=0x7f2a2406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 -- 172.21.15.46:0/3628112288 shutdown_connections 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.750+0000 7f2a2b58e640 1 -- 172.21.15.46:0/3628112288 wait complete. 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.751+0000 7f2a2b58e640 1 Processor -- start 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.751+0000 7f2a2b58e640 1 -- start start 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2b58e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a2406eab0 0x7f2a240cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2b58e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a24146460 0x7f2a240b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2b58e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a240b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2b58e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240ce310 con 0x7f2a24146840 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2b58e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a241f6a50 con 0x7f2a24146460 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2b58e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a241f6d30 con 0x7f2a2406eab0 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a240b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a240b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33174/0 (socket says 172.21.15.46:33174) 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 -- 172.21.15.46:0/669108436 learned_addr learned my addr 172.21.15.46:0/669108436 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2932a640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a2406eab0 0x7f2a240cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a28b29640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a24146460 0x7f2a240b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 -- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a2406eab0 msgr2=0x7f2a240cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a2406eab0 0x7f2a240cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 -- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a24146460 msgr2=0x7f2a240b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a24146460 0x7f2a240b2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 -- 172.21.15.46:0/669108436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180094d0 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a2932a640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a2406eab0 0x7f2a240cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a28b29640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a24146460 0x7f2a240b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.752+0000 7f2a29b2b640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a240b3240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00d580 tx=0x7f2a1c00da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.753+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c014620 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.753+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1c0147a0 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.753+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a241f7010 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.753+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c014b50 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.753+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a241384f0 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.754+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a24145ff0 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.754+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2a1c014cb0 con 0x7f2a24146840 2023-12-16T21:13:51.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.755+0000 7f2a21ffb640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a1006ae40 0x7f2a1006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:51.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.755+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a1c0899c0 con 0x7f2a24146840 2023-12-16T21:13:51.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.755+0000 7f2a2932a640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a1006ae40 0x7f2a1006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:51.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.756+0000 7f2a2932a640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a1006ae40 0x7f2a1006d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400e550 tx=0x7f2a14007ab0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:51.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.759+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2a1c054a80 con 0x7f2a24146840 2023-12-16T21:13:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.935+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2a24069400 con 0x7f2a24146840 2023-12-16T21:13:51.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.935+0000 7f2a21ffb640 1 -- 172.21.15.46:0/669108436 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2a1c0580a0 con 0x7f2a24146840 2023-12-16T21:13:51.935 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a1006ae40 msgr2=0x7f2a1006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a1006ae40 0x7f2a1006d300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400e550 tx=0x7f2a14007ab0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 msgr2=0x7f2a240b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a240b3240 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00d580 tx=0x7f2a1c00da40 comp rx=0 tx=0).stop 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 shutdown_connections 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a1006ae40 0x7f2a1006d300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a24146840 0x7f2a240b3240 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a24146460 0x7f2a240b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 --2- 172.21.15.46:0/669108436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a2406eab0 0x7f2a240cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:51.937 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 >> 172.21.15.46:0/669108436 conn(0x7f2a2406a6b0 msgr2=0x7f2a2406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 shutdown_connections 2023-12-16T21:13:51.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:51.937+0000 7f2a2b58e640 1 -- 172.21.15.46:0/669108436 wait complete. 2023-12-16T21:13:51.947 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:13:52.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.053+0000 7f3c8c59e640 1 Processor -- start 2023-12-16T21:13:52.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.054+0000 7f3c8c59e640 1 -- start start 2023-12-16T21:13:52.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.054+0000 7f3c8c59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c84160fb0 0x7f3c84161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c8c59e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c84161d20 con 0x7f3c8406ea50 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c8c59e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8415e760 con 0x7f3c840d0ed0 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c8c59e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8415d9a0 con 0x7f3c84160fb0 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c89b39640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3c84160fb0 0x7f3c84161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c8a33a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3c8406ea50 0x7f3c8406ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33404/0 (socket says 172.21.15.46:33404) 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c8a33a640 1 -- 172.21.15.46:0/2270580345 learned_addr learned my addr 172.21.15.46:0/2270580345 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c89b39640 1 -- 172.21.15.46:0/2270580345 >> v1:172.21.15.121:6789/0 conn(0x7f3c840d0ed0 legacy=0x7f3c840d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c89b39640 1 -- 172.21.15.46:0/2270580345 >> v1:172.21.15.46:6789/0 conn(0x7f3c8406ea50 legacy=0x7f3c8406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c89b39640 1 -- 172.21.15.46:0/2270580345 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8415dc80 con 0x7f3c84160fb0 2023-12-16T21:13:52.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.055+0000 7f3c89338640 1 -- 172.21.15.46:0/2270580345 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414910955 0 0) 0x7f3c84161d20 con 0x7f3c8406ea50 2023-12-16T21:13:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c89b39640 1 --2- 172.21.15.46:0/2270580345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c84161440 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c740097a0 tx=0x7f3c7402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cb521d25ce5c83 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c89338640 1 -- 172.21.15.46:0/2270580345 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7402e070 con 0x7f3c84160fb0 2023-12-16T21:13:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c89338640 1 -- 172.21.15.46:0/2270580345 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c740315a0 con 0x7f3c84160fb0 2023-12-16T21:13:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c89338640 1 -- 172.21.15.46:0/2270580345 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c74031930 con 0x7f3c84160fb0 2023-12-16T21:13:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c8c59e640 1 -- 172.21.15.46:0/2270580345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 msgr2=0x7f3c84161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/2270580345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c84161440 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c740097a0 tx=0x7f3c7402b840 comp rx=0 tx=0).stop 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c8c59e640 1 -- 172.21.15.46:0/2270580345 shutdown_connections 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/2270580345 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c84161440 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.056+0000 7f3c8c59e640 1 -- 172.21.15.46:0/2270580345 >> 172.21.15.46:0/2270580345 conn(0x7f3c8406a6b0 msgr2=0x7f3c8406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.057+0000 7f3c8c59e640 1 -- 172.21.15.46:0/2270580345 shutdown_connections 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.057+0000 7f3c8c59e640 1 -- 172.21.15.46:0/2270580345 wait complete. 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 Processor -- start 2023-12-16T21:13:52.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 -- start start 2023-12-16T21:13:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c8406ea50 0x7f3c8415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c840d0ed0 0x7f3c8413f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c840cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8415b140 con 0x7f3c84160fb0 2023-12-16T21:13:52.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c840ca7a0 con 0x7f3c8406ea50 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.058+0000 7f3c8c59e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c840caa80 con 0x7f3c840d0ed0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c840cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c840cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33188/0 (socket says 172.21.15.46:33188) 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 -- 172.21.15.46:0/859740192 learned_addr learned my addr 172.21.15.46:0/859740192 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8a33a640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c8406ea50 0x7f3c8415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c89b39640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c840d0ed0 0x7f3c8413f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 -- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c840d0ed0 msgr2=0x7f3c8413f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c840d0ed0 0x7f3c8413f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 -- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c8406ea50 msgr2=0x7f3c8415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c8406ea50 0x7f3c8415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 -- 172.21.15.46:0/859740192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c74009400 con 0x7f3c84160fb0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8ab3b640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c840cd7d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00c780 tx=0x7f3c7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c004b10 con 0x7f3c84160fb0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c004c70 con 0x7f3c84160fb0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c00aa90 con 0x7f3c84160fb0 2023-12-16T21:13:52.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c840cad60 con 0x7f3c84160fb0 2023-12-16T21:13:52.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.059+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c840cb320 con 0x7f3c84160fb0 2023-12-16T21:13:52.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.060+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c840cb510 con 0x7f3c84160fb0 2023-12-16T21:13:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.061+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3c7c00abf0 con 0x7f3c84160fb0 2023-12-16T21:13:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.061+0000 7f3c7affd640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3c6806ae40 0x7f3c6806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.061+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c7c089810 con 0x7f3c84160fb0 2023-12-16T21:13:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.061+0000 7f3c8a33a640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3c6806ae40 0x7f3c6806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.062+0000 7f3c8a33a640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3c6806ae40 0x7f3c6806d300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3c80000f60 tx=0x7f3c80007c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.065+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3c7c054790 con 0x7f3c84160fb0 2023-12-16T21:13:52.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.282+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c84069400 con 0x7f3c84160fb0 2023-12-16T21:13:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.282+0000 7f3c7affd640 1 -- 172.21.15.46:0/859740192 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f3c7c057db0 con 0x7f3c84160fb0 2023-12-16T21:13:52.282 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:52.282 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:13:52.282 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:13:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.284+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3c6806ae40 msgr2=0x7f3c6806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.284+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3c6806ae40 0x7f3c6806d300 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3c80000f60 tx=0x7f3c80007c00 comp rx=0 tx=0).stop 2023-12-16T21:13:52.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.284+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 msgr2=0x7f3c840cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.284+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c840cd7d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c00c780 tx=0x7f3c7c00cc40 comp rx=0 tx=0).stop 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 shutdown_connections 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3c6806ae40 0x7f3c6806d300 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3c84160fb0 0x7f3c840cd7d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c840d0ed0 0x7f3c8413f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 --2- 172.21.15.46:0/859740192 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c8406ea50 0x7f3c8415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 >> 172.21.15.46:0/859740192 conn(0x7f3c8406a6b0 msgr2=0x7f3c8415ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 shutdown_connections 2023-12-16T21:13:52.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.285+0000 7f3c8c59e640 1 -- 172.21.15.46:0/859740192 wait complete. 2023-12-16T21:13:52.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:13:52.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.401+0000 7f1cac743640 1 Processor -- start 2023-12-16T21:13:52.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.401+0000 7f1cac743640 1 -- start start 2023-12-16T21:13:52.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1cac743640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca4140310 0x7f1ca41618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1cac743640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ca41409d0 0x7f1ca406a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1cac743640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca41406f0 con 0x7f1ca41428d0 2023-12-16T21:13:52.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1cac743640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca4161e20 con 0x7f1ca41409d0 2023-12-16T21:13:52.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1cac743640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca4143960 con 0x7f1ca4140310 2023-12-16T21:13:52.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1ca9cde640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1ca41409d0 0x7f1ca406a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1caace0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1ca41428d0 0x7f1ca40714a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33412/0 (socket says 172.21.15.46:33412) 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1caace0640 1 -- 172.21.15.46:0/3901688137 learned_addr learned my addr 172.21.15.46:0/3901688137 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.402+0000 7f1caa4df640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca4140310 0x7f1ca41618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1ca9cde640 1 -- 172.21.15.46:0/3901688137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca4140310 msgr2=0x7f1ca41618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1ca9cde640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca4140310 0x7f1ca41618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1ca9cde640 1 -- 172.21.15.46:0/3901688137 >> v1:172.21.15.46:6789/0 conn(0x7f1ca41428d0 legacy=0x7f1ca40714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1ca9cde640 1 -- 172.21.15.46:0/3901688137 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4142de0 con 0x7f1ca41409d0 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1ca94dd640 1 -- 172.21.15.46:0/3901688137 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682088069 0 0) 0x7f1ca41406f0 con 0x7f1ca41428d0 2023-12-16T21:13:52.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1caa4df640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca4140310 0x7f1ca41618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:52.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.403+0000 7f1ca9cde640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41409d0 0x7f1ca406a4d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1c980097a0 tx=0x7f1c9802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2608196ba35a05e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1ca94dd640 1 -- 172.21.15.46:0/3901688137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9802e070 con 0x7f1ca41409d0 2023-12-16T21:13:52.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1ca94dd640 1 -- 172.21.15.46:0/3901688137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c980315a0 con 0x7f1ca41409d0 2023-12-16T21:13:52.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1ca94dd640 1 -- 172.21.15.46:0/3901688137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98031930 con 0x7f1ca41409d0 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 -- 172.21.15.46:0/3901688137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41409d0 msgr2=0x7f1ca406a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41409d0 0x7f1ca406a4d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1c980097a0 tx=0x7f1c9802b840 comp rx=0 tx=0).stop 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 -- 172.21.15.46:0/3901688137 shutdown_connections 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41409d0 0x7f1ca406a4d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 --2- 172.21.15.46:0/3901688137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca4140310 0x7f1ca41618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 -- 172.21.15.46:0/3901688137 >> 172.21.15.46:0/3901688137 conn(0x7f1ca40b1e50 msgr2=0x7f1ca40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 -- 172.21.15.46:0/3901688137 shutdown_connections 2023-12-16T21:13:52.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.404+0000 7f1cac743640 1 -- 172.21.15.46:0/3901688137 wait complete. 2023-12-16T21:13:52.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.405+0000 7f1cac743640 1 Processor -- start 2023-12-16T21:13:52.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.405+0000 7f1cac743640 1 -- start start 2023-12-16T21:13:52.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1cac743640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca4140310 0x7f1ca4143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1cac743640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca41409d0 0x7f1ca413d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1cac743640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 0x7f1ca40b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1cac743640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca40b1280 con 0x7f1ca41428d0 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1cac743640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41441c0 con 0x7f1ca4140310 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1cac743640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca41444a0 con 0x7f1ca41409d0 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1caace0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 0x7f1ca40b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1caace0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 0x7f1ca40b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33220/0 (socket says 172.21.15.46:33220) 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1caace0640 1 -- 172.21.15.46:0/3675441175 learned_addr learned my addr 172.21.15.46:0/3675441175 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.406+0000 7f1caa4df640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca4140310 0x7f1ca4143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1ca9cde640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca41409d0 0x7f1ca413d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1caace0640 1 -- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca41409d0 msgr2=0x7f1ca413d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1caace0640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca41409d0 0x7f1ca413d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1caace0640 1 -- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca4140310 msgr2=0x7f1ca4143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1caace0640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca4140310 0x7f1ca4143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1caace0640 1 -- 172.21.15.46:0/3675441175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c98009400 con 0x7f1ca41428d0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1caace0640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 0x7f1ca40b3f90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c7a0 tx=0x7f1c9400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9400a640 con 0x7f1ca41428d0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9400a7a0 con 0x7f1ca41428d0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca4144780 con 0x7f1ca41428d0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9400abd0 con 0x7f1ca41428d0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.407+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca413de30 con 0x7f1ca41428d0 2023-12-16T21:13:52.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.408+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca4144ca0 con 0x7f1ca41428d0 2023-12-16T21:13:52.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.408+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1c9400ad30 con 0x7f1ca41428d0 2023-12-16T21:13:52.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.409+0000 7f1ca2ffd640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c9006ae40 0x7f1c9006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.409+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c94088e30 con 0x7f1ca41428d0 2023-12-16T21:13:52.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.409+0000 7f1caa4df640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c9006ae40 0x7f1c9006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.409+0000 7f1caa4df640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c9006ae40 0x7f1c9006d300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c006fd0 tx=0x7f1c9c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.413+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1ca4144ca0 con 0x7f1ca41428d0 2023-12-16T21:13:52.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.593+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ca4144ca0 con 0x7f1ca41428d0 2023-12-16T21:13:52.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.594+0000 7f1ca2ffd640 1 -- 172.21.15.46:0/3675441175 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1ca4144ca0 con 0x7f1ca41428d0 2023-12-16T21:13:52.593 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:52.594 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:13:52.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c9006ae40 msgr2=0x7f1c9006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c9006ae40 0x7f1c9006d300 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c006fd0 tx=0x7f1c9c008040 comp rx=0 tx=0).stop 2023-12-16T21:13:52.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 msgr2=0x7f1ca40b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 0x7f1ca40b3f90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400c7a0 tx=0x7f1c9400cc60 comp rx=0 tx=0).stop 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 shutdown_connections 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c9006ae40 0x7f1c9006d300 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ca41428d0 0x7f1ca40b3f90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ca41409d0 0x7f1ca413d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 --2- 172.21.15.46:0/3675441175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ca4140310 0x7f1ca4143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 >> 172.21.15.46:0/3675441175 conn(0x7f1ca40b1e50 msgr2=0x7f1ca406f820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 shutdown_connections 2023-12-16T21:13:52.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.596+0000 7f1cac743640 1 -- 172.21.15.46:0/3675441175 wait complete. 2023-12-16T21:13:52.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:13:52.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 Processor -- start 2023-12-16T21:13:52.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 -- start start 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18200cf320 0x7f18200d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015f760 0x7f18200cd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18200b3f10 con 0x7f182015b130 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182015fd20 con 0x7f18200cf320 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.715+0000 7f1826e51640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18200d0730 con 0x7f182015f760 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18200cf320 0x7f18200d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f18200cf320 0x7f18200d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33222/0 (socket says 172.21.15.46:33222) 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 -- 172.21.15.46:0/1224827268 learned_addr learned my addr 172.21.15.46:0/1224827268 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f18253ee640 1 --2- 172.21.15.46:0/1224827268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015f760 0x7f18200cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 -- 172.21.15.46:0/1224827268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015f760 msgr2=0x7f18200cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 --2- 172.21.15.46:0/1224827268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015f760 0x7f18200cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 -- 172.21.15.46:0/1224827268 >> v1:172.21.15.121:6789/0 conn(0x7f182015b130 legacy=0x7f18200cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.716+0000 7f181ffff640 1 -- 172.21.15.46:0/1224827268 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18200d0a50 con 0x7f18200cf320 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f181ffff640 1 --2- 172.21.15.46:0/1224827268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18200cf320 0x7f18200d01f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1814009870 tx=0x7f181402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa050134d53e17a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f181f7fe640 1 -- 172.21.15.46:0/1224827268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814030840 con 0x7f18200cf320 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f181f7fe640 1 -- 172.21.15.46:0/1224827268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18140309a0 con 0x7f18200cf320 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f181f7fe640 1 -- 172.21.15.46:0/1224827268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814030d30 con 0x7f18200cf320 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 -- 172.21.15.46:0/1224827268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18200cf320 msgr2=0x7f18200d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 --2- 172.21.15.46:0/1224827268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18200cf320 0x7f18200d01f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1814009870 tx=0x7f181402b980 comp rx=0 tx=0).stop 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 -- 172.21.15.46:0/1224827268 shutdown_connections 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 --2- 172.21.15.46:0/1224827268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015f760 0x7f18200cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 --2- 172.21.15.46:0/1224827268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f18200cf320 0x7f18200d01f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 -- 172.21.15.46:0/1224827268 >> 172.21.15.46:0/1224827268 conn(0x7f18200b2c10 msgr2=0x7f18200b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 -- 172.21.15.46:0/1224827268 shutdown_connections 2023-12-16T21:13:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.717+0000 7f1826e51640 1 -- 172.21.15.46:0/1224827268 wait complete. 2023-12-16T21:13:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.719+0000 7f1826e51640 1 Processor -- start 2023-12-16T21:13:52.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.719+0000 7f1826e51640 1 -- start start 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.719+0000 7f1826e51640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18200cf320 0x7f182014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.719+0000 7f1826e51640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015b130 0x7f18200c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.719+0000 7f1826e51640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 0x7f18200c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.719+0000 7f1826e51640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201f4b30 con 0x7f182015f760 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f1826e51640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201f3eb0 con 0x7f18200cf320 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f1826e51640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18201f4190 con 0x7f182015b130 2023-12-16T21:13:52.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 0x7f18200c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 0x7f18200c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33238/0 (socket says 172.21.15.46:33238) 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 -- 172.21.15.46:0/4111033367 learned_addr learned my addr 172.21.15.46:0/4111033367 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f181ffff640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015b130 0x7f18200c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f1824bed640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18200cf320 0x7f182014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 -- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015b130 msgr2=0x7f18200c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015b130 0x7f18200c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 -- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18200cf320 msgr2=0x7f182014d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18200cf320 0x7f182014d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 -- 172.21.15.46:0/4111033367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140094d0 con 0x7f182015f760 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f18253ee640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 0x7f18200c90d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f181000e7f0 tx=0x7f181000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810009510 con 0x7f182015f760 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181000ee40 con 0x7f182015f760 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181000cb80 con 0x7f182015f760 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18201f4470 con 0x7f182015f760 2023-12-16T21:13:52.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.720+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18200c8650 con 0x7f182015f760 2023-12-16T21:13:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.721+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1820067fc0 con 0x7f182015f760 2023-12-16T21:13:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.722+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f181000cce0 con 0x7f182015f760 2023-12-16T21:13:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.722+0000 7f181d7fa640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f180c06ae40 0x7f180c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:52.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.722+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1810089630 con 0x7f182015f760 2023-12-16T21:13:52.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.722+0000 7f1824bed640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f180c06ae40 0x7f180c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:52.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.723+0000 7f1824bed640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f180c06ae40 0x7f180c06d300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f18180012e0 tx=0x7f1818009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:52.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.726+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1810054780 con 0x7f182015f760 2023-12-16T21:13:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.938+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18201f4c70 con 0x7f182015f760 2023-12-16T21:13:52.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.938+0000 7f181d7fa640 1 -- 172.21.15.46:0/4111033367 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f18201f4c70 con 0x7f182015f760 2023-12-16T21:13:52.938 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:13:52.938 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:13:52.939 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f180c06ae40 msgr2=0x7f180c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f180c06ae40 0x7f180c06d300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f18180012e0 tx=0x7f1818009040 comp rx=0 tx=0).stop 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 msgr2=0x7f18200c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 0x7f18200c90d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f181000e7f0 tx=0x7f181000ebf0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 shutdown_connections 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f180c06ae40 0x7f180c06d300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f182015f760 0x7f18200c90d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f182015b130 0x7f18200c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 --2- 172.21.15.46:0/4111033367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18200cf320 0x7f182014d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 >> 172.21.15.46:0/4111033367 conn(0x7f18200b2c10 msgr2=0x7f182006b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 shutdown_connections 2023-12-16T21:13:52.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:52.940+0000 7f1826e51640 1 -- 172.21.15.46:0/4111033367 wait complete. 2023-12-16T21:13:53.353 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-16T21:13:35.861120+0000', 'modified': '2023-12-16T21:13:37.862845+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6837/1967157376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6835', 'nonce': 1967157376}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 1967157376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-16T21:13:53.353 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-16T21:13:53.354 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-16T21:13:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.464+0000 7f906f94c640 1 Processor -- start 2023-12-16T21:13:53.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.464+0000 7f906f94c640 1 -- start start 2023-12-16T21:13:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906f94c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9068143490 0x7f90680b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906f94c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 0x7f90680b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906f94c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f906815e2a0 con 0x7f906815a4e0 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906f94c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f906815b520 con 0x7f9068143490 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906f94c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9068143870 con 0x7f906815e730 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906cee7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 0x7f90680b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906d6e8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9068143490 0x7f90680b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906cee7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 0x7f90680b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:41504/0 (socket says 172.21.15.46:41504) 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906dee9640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f906815a4e0 0x7f906815a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57226/0 (socket says 172.21.15.46:57226) 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906cee7640 1 -- 172.21.15.46:0/2577919600 learned_addr learned my addr 172.21.15.46:0/2577919600 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.465+0000 7f906d6e8640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9068143490 0x7f90680b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:57410/0 (socket says 172.21.15.46:57410) 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.466+0000 7f906d6e8640 1 -- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 msgr2=0x7f90680b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.466+0000 7f906d6e8640 1 --2- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 0x7f90680b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.466+0000 7f906d6e8640 1 -- 172.21.15.46:0/2577919600 >> v1:172.21.15.121:6789/0 conn(0x7f906815a4e0 legacy=0x7f906815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.466+0000 7f906d6e8640 1 -- 172.21.15.46:0/2577919600 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906815b240 con 0x7f9068143490 2023-12-16T21:13:53.466 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.466+0000 7f906cee7640 1 --2- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 0x7f90680b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:13:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906d6e8640 1 --2- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9068143490 0x7f90680b0cf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f90600097a0 tx=0x7f906002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bc93604beceed12 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f9067fff640 1 -- 172.21.15.46:0/2577919600 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060031830 con 0x7f9068143490 2023-12-16T21:13:53.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f9067fff640 1 -- 172.21.15.46:0/2577919600 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9060031990 con 0x7f9068143490 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f9067fff640 1 -- 172.21.15.46:0/2577919600 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9060031d20 con 0x7f9068143490 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 -- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9068143490 msgr2=0x7f90680b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 --2- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9068143490 0x7f90680b0cf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f90600097a0 tx=0x7f906002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 -- 172.21.15.46:0/2577919600 shutdown_connections 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 --2- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815e730 0x7f90680b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 --2- 172.21.15.46:0/2577919600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9068143490 0x7f90680b0cf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 -- 172.21.15.46:0/2577919600 >> 172.21.15.46:0/2577919600 conn(0x7f906806a4d0 msgr2=0x7f906806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 -- 172.21.15.46:0/2577919600 shutdown_connections 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.467+0000 7f906f94c640 1 -- 172.21.15.46:0/2577919600 wait complete. 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.468+0000 7f906f94c640 1 Processor -- start 2023-12-16T21:13:53.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 -- start start 2023-12-16T21:13:53.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 0x7f90680ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815a4e0 0x7f90680b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f906815e730 0x7f906815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90680b2390 con 0x7f9068143490 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90680acf70 con 0x7f906815e730 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906f94c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90680ad250 con 0x7f906815a4e0 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906d6e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 0x7f90680ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906d6e8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 0x7f90680ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33242/0 (socket says 172.21.15.46:33242) 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906d6e8640 1 -- 172.21.15.46:0/3945882367 learned_addr learned my addr 172.21.15.46:0/3945882367 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906dee9640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f906815e730 0x7f906815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.469+0000 7f906cee7640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815a4e0 0x7f90680b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906d6e8640 1 -- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815a4e0 msgr2=0x7f90680b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906d6e8640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815a4e0 0x7f90680b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906d6e8640 1 -- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f906815e730 msgr2=0x7f906815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906d6e8640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f906815e730 0x7f906815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906d6e8640 1 -- 172.21.15.46:0/3945882367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905c0094d0 con 0x7f9068143490 2023-12-16T21:13:53.472 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906d6e8640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 0x7f90680ac8e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f90600098c0 tx=0x7f90600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f9065ffb640 1 -- 172.21.15.46:0/3945882367 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906003a6f0 con 0x7f9068143490 2023-12-16T21:13:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f9065ffb640 1 -- 172.21.15.46:0/3945882367 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f906003a870 con 0x7f9068143490 2023-12-16T21:13:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9060009400 con 0x7f9068143490 2023-12-16T21:13:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f9065ffb640 1 -- 172.21.15.46:0/3945882367 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906003ac20 con 0x7f9068143490 2023-12-16T21:13:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.470+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90680ad810 con 0x7f9068143490 2023-12-16T21:13:53.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.472+0000 7f9065ffb640 1 -- 172.21.15.46:0/3945882367 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f906004c3b0 con 0x7f9068143490 2023-12-16T21:13:53.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.472+0000 7f9065ffb640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f905406ae40 0x7f905406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.472+0000 7f9065ffb640 1 -- 172.21.15.46:0/3945882367 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90600ab1d0 con 0x7f9068143490 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.473+0000 7f906cee7640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f905406ae40 0x7f905406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.473+0000 7f906cee7640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f905406ae40 0x7f905406d300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f90681e7740 tx=0x7f905c009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.473+0000 7f90725c0740 1 Processor -- start 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.473+0000 7f90725c0740 1 -- start start 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b07373b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0d40 0x561b073a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f90725c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 0x561b073a3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f90725c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b072e7090 con 0x561b073740d0 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b072e4780 con 0x561b073a0d40 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b072e0070 con 0x561b073a0960 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 0x561b073a3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 0x561b073a3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33258/0 (socket says 172.21.15.46:33258) 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 -- 172.21.15.46:0/2663631169 learned_addr learned my addr 172.21.15.46:0/2663631169 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906d6e8640 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b07373b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906cee7640 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0d40 0x561b073a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 -- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 msgr2=0x561b07373b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b07373b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 -- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0d40 msgr2=0x561b073a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0d40 0x561b073a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.474+0000 7f906dee9640 1 -- 172.21.15.46:0/2663631169 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b072a2d00 con 0x561b073740d0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f906dee9640 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 0x561b073a3c30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9058001980 tx=0x7f9058001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2624c6c3d858296f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f904effd640 1 -- 172.21.15.46:0/2663631169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058002090 con 0x561b073740d0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f904effd640 1 -- 172.21.15.46:0/2663631169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9058007850 con 0x561b073740d0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f904effd640 1 -- 172.21.15.46:0/2663631169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580133f0 con 0x561b073740d0 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 -- 172.21.15.46:0/2663631169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 msgr2=0x561b073a3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 0x561b073a3c30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9058001980 tx=0x7f9058001e40 comp rx=0 tx=0).stop 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 -- 172.21.15.46:0/2663631169 shutdown_connections 2023-12-16T21:13:53.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073740d0 0x561b073a3c30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0d40 0x561b073a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 --2- 172.21.15.46:0/2663631169 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b07373b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 -- 172.21.15.46:0/2663631169 >> 172.21.15.46:0/2663631169 conn(0x561b07375e00 msgr2=0x561b073be6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 -- 172.21.15.46:0/2663631169 shutdown_connections 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.475+0000 7f90725c0740 1 -- 172.21.15.46:0/2663631169 wait complete. 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.476+0000 7f90725c0740 1 -- start start 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.476+0000 7f90725c0740 1 Processor -- start 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b073aa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073a8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a8560 0x561b0738f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b07299e70 con 0x561b073a0d40 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b07299b10 con 0x561b073a8560 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b0725f2e0 con 0x561b073a0960 2023-12-16T21:13:53.481 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f90725c0740 10 client.0 _ll_register_callbacks cb 0x561b06ff1ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073a8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073a8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33270/0 (socket says 172.21.15.46:33270) 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 -- 172.21.15.46:0/702008472 learned_addr learned my addr 172.21.15.46:0/702008472 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906d6e8640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b073aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906dee9640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a8560 0x561b0738f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 -- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 msgr2=0x561b073aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b073aa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 -- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a8560 msgr2=0x561b0738f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a8560 0x561b0738f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.477+0000 7f906cee7640 1 -- 172.21.15.46:0/702008472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906815e2a0 con 0x561b073a0d40 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f906cee7640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073a8020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f905c014e00 tx=0x7f905c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e81dbc9e525fcf server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f9047fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f9047fff640 1 -- 172.21.15.46:0/702008472 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c013920 con 0x561b073a0d40 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f9047fff640 1 -- 172.21.15.46:0/702008472 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f905c01c040 con 0x561b073a0d40 2023-12-16T21:13:53.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b072d8f00 con 0x561b073a0d40 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f9047fff640 1 -- 172.21.15.46:0/702008472 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f905c013a80 con 0x561b073a0d40 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 10 client.4668 fetch_fsmap learned FSMap version 6 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x561b072de6f0 con 0x561b073a0d40 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f9047fff640 1 -- 172.21.15.46:0/702008472 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b072d8f00 con 0x561b073a0d40 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f9047fff640 1 -- 172.21.15.46:0/702008472 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 6) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f905c013be0 con 0x561b073a0d40 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 10 client.4668 fetch_fsmap finished waiting for FSMap version 6 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 10 client.4668 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 20 client.4668 populate_metadata read hostname 'smithi046' 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a4bc0 0x561b073a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 4 client.4668 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 --> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b072d8c20 con 0x561b073a4bc0 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.478+0000 7f906d6e8640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a4bc0 0x561b073a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.479+0000 7f906d6e8640 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a4bc0 0x561b073a4fa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=586e1ef0e9aba5f8 server_cookie=3db7e5ecf1d1801d in_seq=0 out_seq=0 2023-12-16T21:13:53.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.479+0000 7f9047fff640 10 client.4668 ms_handle_connect on v2:172.21.15.121:6835/1967157376 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.479+0000 7f9047fff640 1 -- 172.21.15.46:0/702008472 <== mds.0 v2:172.21.15.121:6835/1967157376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b072d8c20 con 0x561b073a4bc0 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.479+0000 7f9047fff640 10 client.4668 handle_command_reply: tid=0 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.479+0000 7f90725c0740 1 client.4668 shutdown 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 msgr2=0x561b073a8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073a8020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f905c014e00 tx=0x7f905c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 shutdown_connections 2023-12-16T21:13:53.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a4bc0 0x561b073a4fa0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073a8020 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f9047fff640 0 client.4668 ms_handle_reset on v2:172.21.15.121:6835/1967157376 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a8560 0x561b0738f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 --2- 172.21.15.46:0/702008472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0960 0x561b073aa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 >> 172.21.15.46:0/702008472 conn(0x561b073740d0 msgr2=0x561b073a36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 shutdown_connections 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 1 -- 172.21.15.46:0/702008472 wait complete. 2023-12-16T21:13:53.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.480+0000 7f90725c0740 20 client.4668 trim_cache size 0 max 16384 2023-12-16T21:13:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.495+0000 7f90725c0740 1 Processor -- start 2023-12-16T21:13:53.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.495+0000 7f90725c0740 1 -- start start 2023-12-16T21:13:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b073e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f90725c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073aa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a8560 0x561b073bf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f90725c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b07372be0 con 0x561b073a0d40 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b073e09a0 con 0x561b073a0960 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b07377ea0 con 0x561b073a8560 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073aa100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33272/0 (socket says 172.21.15.46:33272) 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 -- 172.21.15.46:0/433456003 learned_addr learned my addr 172.21.15.46:0/433456003 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906dee9640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a8560 0x561b073bf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906d6e8640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b073e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 -- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a8560 msgr2=0x561b073bf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a8560 0x561b073bf940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 -- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 msgr2=0x561b073e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b073e0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906cee7640 1 -- 172.21.15.46:0/433456003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b0727bd70 con 0x561b073a0d40 2023-12-16T21:13:53.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906dee9640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a8560 0x561b073bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.496+0000 7f906d6e8640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b073e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f906cee7640 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073aa100 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f905c011ff0 tx=0x7f905c01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd6871da72a3897 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f904f7fe640 1 -- 172.21.15.46:0/433456003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c011b70 con 0x561b073a0d40 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f904f7fe640 1 -- 172.21.15.46:0/433456003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c023480 con 0x561b073a0d40 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f904f7fe640 1 -- 172.21.15.46:0/433456003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c013040 con 0x561b073a0d40 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 -- 172.21.15.46:0/433456003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 msgr2=0x561b073aa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073aa100 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f905c011ff0 tx=0x7f905c01b000 comp rx=0 tx=0).stop 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 -- 172.21.15.46:0/433456003 shutdown_connections 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a0d40 0x561b073aa100 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a8560 0x561b073bf940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 --2- 172.21.15.46:0/433456003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b073e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 -- 172.21.15.46:0/433456003 >> 172.21.15.46:0/433456003 conn(0x561b073740d0 msgr2=0x561b073c1150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 -- 172.21.15.46:0/433456003 shutdown_connections 2023-12-16T21:13:53.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.497+0000 7f90725c0740 1 -- 172.21.15.46:0/433456003 wait complete. 2023-12-16T21:13:53.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.498+0000 7f90725c0740 1 -- start start 2023-12-16T21:13:53.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.498+0000 7f90725c0740 1 Processor -- start 2023-12-16T21:13:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b0739ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0d40 0x561b0739f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 0x561b0739f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b073d63e0 con 0x561b073a8560 2023-12-16T21:13:53.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b073d3330 con 0x561b073a0960 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b073c26d0 con 0x561b073a0d40 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f90725c0740 10 client.0 _ll_register_callbacks cb 0x7f9058004c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 0x561b0739f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 0x561b0739f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33276/0 (socket says 172.21.15.46:33276) 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 -- 172.21.15.46:0/2632196007 learned_addr learned my addr 172.21.15.46:0/2632196007 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906d6e8640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b0739ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906cee7640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0d40 0x561b0739f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 -- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0d40 msgr2=0x561b0739f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0d40 0x561b0739f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 -- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 msgr2=0x561b0739ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b0739ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 -- 172.21.15.46:0/2632196007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b072de6f0 con 0x561b073a8560 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f906dee9640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 0x561b0739f750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9058001950 tx=0x7f9058012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbde8088aa1a1080 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f9047fff640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-12-16T21:13:53.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.499+0000 7f9047fff640 1 -- 172.21.15.46:0/2632196007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9058024070 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f9047fff640 1 -- 172.21.15.46:0/2632196007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9058002090 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b07373930 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f9047fff640 1 -- 172.21.15.46:0/2632196007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f905801f070 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 10 client.4674 fetch_fsmap learned FSMap version 6 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x561b072a1560 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f9047fff640 1 -- 172.21.15.46:0/2632196007 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561b07373930 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f9047fff640 1 -- 172.21.15.46:0/2632196007 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 6) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9058012070 con 0x561b073a8560 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 10 client.4674 fetch_fsmap finished waiting for FSMap version 6 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 10 client.4674 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 20 client.4674 populate_metadata read hostname 'smithi046' 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a36d0 0x561b073a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 4 client.4674 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-16T21:13:53.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 --> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f9058007850 con 0x561b073a36d0 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f906d6e8640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a36d0 0x561b073a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f906d6e8640 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a36d0 0x561b073a3ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d71eeea90321342 server_cookie=ab224b6e0485d487 in_seq=0 out_seq=0 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.500+0000 7f9047fff640 10 client.4674 ms_handle_connect on v2:172.21.15.121:6835/1967157376 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f9047fff640 1 -- 172.21.15.46:0/2632196007 <== mds.0 v2:172.21.15.121:6835/1967157376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7f9058007850 con 0x561b073a36d0 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f9047fff640 10 client.4674 handle_command_reply: tid=0 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 client.4674 shutdown 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 msgr2=0x561b0739f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 0x561b0739f750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9058001950 tx=0x7f9058012000 comp rx=0 tx=0).stop 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 shutdown_connections 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:6835/1967157376,v1:172.21.15.121:6837/1967157376] conn(0x561b073a36d0 0x561b073a3ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b073a8560 0x561b0739f750 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561b073a0d40 0x561b0739f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 --2- 172.21.15.46:0/2632196007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561b073a0960 0x561b0739ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 >> 172.21.15.46:0/2632196007 conn(0x561b073740d0 msgr2=0x561b073a6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f9047fff640 0 client.4674 ms_handle_reset on v2:172.21.15.121:6835/1967157376 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 shutdown_connections 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 1 -- 172.21.15.46:0/2632196007 wait complete. 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.501+0000 7f90725c0740 20 client.4674 trim_cache size 0 max 16384 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "2397e6ee-7ffe-4438-b275-b4e10dea2d2f", 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4294, 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2023-12-16T21:13:53.507 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 16.988846445, 2023-12-16T21:13:53.508 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 6, 2023-12-16T21:13:53.508 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 21, 2023-12-16T21:13:53.508 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 21, 2023-12-16T21:13:53.508 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 23.954533599000001 2023-12-16T21:13:53.508 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:13:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f905406ae40 msgr2=0x7f905406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f905406ae40 0x7f905406d300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f90681e7740 tx=0x7f905c009240 comp rx=0 tx=0).stop 2023-12-16T21:13:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 msgr2=0x7f90680ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 0x7f90680ac8e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f90600098c0 tx=0x7f90600046c0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 shutdown_connections 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f905406ae40 0x7f905406d300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f906815e730 0x7f906815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f906815a4e0 0x7f90680b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 --2- 172.21.15.46:0/3945882367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9068143490 0x7f90680ac8e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 >> 172.21.15.46:0/3945882367 conn(0x7f906806a4d0 msgr2=0x7f9068070c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 shutdown_connections 2023-12-16T21:13:53.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.502+0000 7f906f94c640 1 -- 172.21.15.46:0/3945882367 wait complete. 2023-12-16T21:13:53.512 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-12-16T21:13:53.540 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-12-16T21:13:53.540 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-12-16T21:13:53.540 INFO:tasks.ceph_fuse:client.0 config is {} 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:13:53.541 INFO:tasks.ceph_fuse:client.2 config is {} 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-16T21:13:53.541 INFO:tasks.ceph_fuse:client.1 config is {} 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T21:13:53.541 INFO:tasks.ceph_fuse:client.3 config is {} 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:13:53.541 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-16T21:13:53.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:53.542 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:13:53.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:53.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2023-12-16T21:13:53.630 INFO:teuthology.orchestra.run.smithi046.stderr:Cannot find device "ceph-brx" 2023-12-16T21:13:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:13:53.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:53.631 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2023-12-16T21:13:53.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:53.651 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2023-12-16T21:13:53.724 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2023-12-16T21:13:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:13:53.725 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-12-16T21:13:53.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:13:53.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.833+0000 7f626499e640 1 Processor -- start 2023-12-16T21:13:53.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.833+0000 7f626499e640 1 -- start start 2023-12-16T21:13:53.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f626499e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f62600b1400 0x7f626015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f626499e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626015e2a0 con 0x7f626015e730 2023-12-16T21:13:53.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f626499e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f626015b240 con 0x7f6260143490 2023-12-16T21:13:53.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f626499e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f626015b520 con 0x7f62600b1400 2023-12-16T21:13:53.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f626015e730 0x7f626015aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33426/0 (socket says 172.21.15.46:33426) 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625ed9d640 1 -- 172.21.15.46:0/3043396652 learned_addr learned my addr 172.21.15.46:0/3043396652 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625dd9b640 1 --2- 172.21.15.46:0/3043396652 >> v2:172.21.15.121:3300/0 conn(0x7f62600b1400 0x7f626015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392404233 0 0) 0x7f626015e2a0 con 0x7f626015e730 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c003520 con 0x7f626015e730 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286200547 0 0) 0x7f626015b240 con 0x7f6260143490 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.834+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f626015e2a0 con 0x7f6260143490 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.835+0000 7f625dd9b640 1 -- 172.21.15.46:0/3043396652 >> v1:172.21.15.121:6789/0 conn(0x7f6260143490 legacy=0x7f62600b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.835+0000 7f625dd9b640 1 -- 172.21.15.46:0/3043396652 >> v1:172.21.15.46:6789/0 conn(0x7f626015e730 legacy=0x7f626015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.835+0000 7f625dd9b640 1 -- 172.21.15.46:0/3043396652 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626015dc50 con 0x7f62600b1400 2023-12-16T21:13:53.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.835+0000 7f625dd9b640 1 --2- 172.21.15.46:0/3043396652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62600b1400 0x7f626015a4e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62480097a0 tx=0x7f624802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a991e00b29d31f4e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248030610 con 0x7f62600b1400 2023-12-16T21:13:53.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6248030770 con 0x7f62600b1400 2023-12-16T21:13:53.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f625d59a640 1 -- 172.21.15.46:0/3043396652 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248030ae0 con 0x7f62600b1400 2023-12-16T21:13:53.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 -- 172.21.15.46:0/3043396652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62600b1400 msgr2=0x7f626015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 --2- 172.21.15.46:0/3043396652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62600b1400 0x7f626015a4e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62480097a0 tx=0x7f624802b960 comp rx=0 tx=0).stop 2023-12-16T21:13:53.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 -- 172.21.15.46:0/3043396652 shutdown_connections 2023-12-16T21:13:53.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 --2- 172.21.15.46:0/3043396652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62600b1400 0x7f626015a4e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 -- 172.21.15.46:0/3043396652 >> 172.21.15.46:0/3043396652 conn(0x7f626006a4d0 msgr2=0x7f626006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:53.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 -- 172.21.15.46:0/3043396652 shutdown_connections 2023-12-16T21:13:53.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.836+0000 7f626499e640 1 -- 172.21.15.46:0/3043396652 wait complete. 2023-12-16T21:13:53.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.837+0000 7f626499e640 1 Processor -- start 2023-12-16T21:13:53.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 -- start start 2023-12-16T21:13:53.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 0x7f6260159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6260143490 0x7f62600b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f626015e730 0x7f6260070ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62600b1290 con 0x7f62600b1400 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626013aeb0 con 0x7f626015e730 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f626499e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f626013b190 con 0x7f6260143490 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f625e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 0x7f6260159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f625e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 0x7f6260159b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33278/0 (socket says 172.21.15.46:33278) 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.838+0000 7f625e59c640 1 -- 172.21.15.46:0/1529137765 learned_addr learned my addr 172.21.15.46:0/1529137765 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625dd9b640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6260143490 0x7f62600b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625ed9d640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f626015e730 0x7f6260070ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625e59c640 1 -- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6260143490 msgr2=0x7f62600b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625e59c640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6260143490 0x7f62600b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625e59c640 1 -- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f626015e730 msgr2=0x7f6260070ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625e59c640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f626015e730 0x7f6260070ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625e59c640 1 -- 172.21.15.46:0/1529137765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6248009400 con 0x7f62600b1400 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f625e59c640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 0x7f6260159b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f625000c870 tx=0x7f625000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250003b70 con 0x7f62600b1400 2023-12-16T21:13:53.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250003cf0 con 0x7f62600b1400 2023-12-16T21:13:53.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625000aa90 con 0x7f62600b1400 2023-12-16T21:13:53.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626013b470 con 0x7f62600b1400 2023-12-16T21:13:53.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.839+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626013ba30 con 0x7f62600b1400 2023-12-16T21:13:53.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.840+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6260069400 con 0x7f62600b1400 2023-12-16T21:13:53.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.841+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f625000abf0 con 0x7f62600b1400 2023-12-16T21:13:53.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.841+0000 7f6256ffd640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f624006ae40 0x7f624006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:53.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.841+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6250089330 con 0x7f62600b1400 2023-12-16T21:13:53.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.841+0000 7f625dd9b640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f624006ae40 0x7f624006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:53.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.842+0000 7f625dd9b640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f624006ae40 0x7f624006d300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6248036000 tx=0x7f6248008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:53.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:53.845+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6250054520 con 0x7f62600b1400 2023-12-16T21:13:54.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.059+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f62600cb1d0 con 0x7f62600b1400 2023-12-16T21:13:54.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.059+0000 7f6256ffd640 1 -- 172.21.15.46:0/1529137765 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6250003e60 con 0x7f62600b1400 2023-12-16T21:13:54.059 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:13:54.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f624006ae40 msgr2=0x7f624006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f624006ae40 0x7f624006d300 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6248036000 tx=0x7f6248008030 comp rx=0 tx=0).stop 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 msgr2=0x7f6260159b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 0x7f6260159b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f625000c870 tx=0x7f625000cd30 comp rx=0 tx=0).stop 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 shutdown_connections 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f624006ae40 0x7f624006d300 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f626015e730 0x7f6260070ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6260143490 0x7f62600b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 --2- 172.21.15.46:0/1529137765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f62600b1400 0x7f6260159b70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 >> 172.21.15.46:0/1529137765 conn(0x7f626006a4d0 msgr2=0x7f62601416b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 shutdown_connections 2023-12-16T21:13:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:54.061+0000 7f626499e640 1 -- 172.21.15.46:0/1529137765 wait complete. 2023-12-16T21:13:54.071 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:13:54.071 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:13:54.071 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:13:54.071 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:13:54.072 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:13:54.072 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:13:54.072 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:13:54.072 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:13:54.072 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:13:54.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.072 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:13:54.087 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:13:54.087 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:13:54.087 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:13:54.087 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:54.087 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:13:54.088 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42651sec preferred_lft 42651sec 2023-12-16T21:13:54.089 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:13:54.089 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:54.089 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:54.089 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:54.089 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:13:54.089 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T21:13:54.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.089 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:54.089 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:54.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2023-12-16T21:13:54.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2023-12-16T21:13:54.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2023-12-16T21:13:54.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T21:13:54.090 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:54.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.265 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:13:54.295 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:13:54.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.297 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:13:54.371 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:13:54.371 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:13:54.371 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:13:54.372 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:13:54.372 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:13:54.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.372 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:54.372 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:54.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:13:54.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:13:54.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:13:54.373 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:54.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.690 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:13:54.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.706 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2023-12-16T21:13:54.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.762 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:54.762 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:54.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:54.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-16T21:13:54.762 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:54.856 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-16T21:13:54.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.856 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:54.856 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:54.856 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-16T21:13:54.856 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T21:13:54.856 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-16T21:13:54.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-16T21:13:54.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-16T21:13:54.857 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:54.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:54.978 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:54.978 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:54.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:13:54.978 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T21:13:54.978 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:55.027 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:13:55.027 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:13:55.027 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:13:55.042 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:13:55.042 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:13:55.042 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:13:55.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:13:55.170 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:13:55.226 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:13:55.226 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:13:55.226 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:13:55.226 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:13:55.227 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:13:55.228 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:13:55.229 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:55.229 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:55.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:55.229 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:13:55.281 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:13:55.281 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:13:55.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:13:55.354 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:13:55.410 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:13:55.410 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:13:55.410 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:13:55.410 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:13:55.410 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:13:55.411 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:55.412 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:55.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:55.413 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:13:55.423 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:13:55.424+0000 7fdc5301d480 -1 init, newargv = 0x55c9035f7bb0 newargc=15 2023-12-16T21:13:55.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[98530]: starting ceph client 2023-12-16T21:13:55.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[98530]: starting fuse 2023-12-16T21:13:55.439 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:13:55.439 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:55.439 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:13:55.440 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:13:55.441 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:13:55.441 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:13:55.441 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:55.441 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:13:55.441 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:13:55.565 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.98530.asok 2023-12-16T21:13:55.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:13:55.575 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.98530.asok 2023-12-16T21:13:55.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:55.576 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.98530.asok status 2023-12-16T21:13:55.728 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:13:55.728 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:13:55.728 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:13:55.728 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "98530", 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4686, 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4686 2023-12-16T21:13:55.729 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2805414584 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2805414584 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4686 192.168.0.1:0/2805414584", 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2805414584", 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:13:55.730 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 6, 2023-12-16T21:13:55.731 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 21, 2023-12-16T21:13:55.731 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:13:55.731 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:13:55.731 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:13:55.731 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:13:55.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:13:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.843+0000 7f70b95e5640 1 Processor -- start 2023-12-16T21:13:55.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.843+0000 7f70b95e5640 1 -- start start 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b95e5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70b406f210 0x7f70b4140800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b95e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70b4140d40 0x7f70b413f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b95e5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b415ba40 con 0x7f70b415c2f0 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b95e5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b415c6d0 con 0x7f70b4140d40 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b95e5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b413e530 con 0x7f70b406f210 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70b4140d40 0x7f70b413f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f70b4140d40 0x7f70b413f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33288/0 (socket says 172.21.15.46:33288) 2023-12-16T21:13:55.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 -- 172.21.15.46:0/903133757 learned_addr learned my addr 172.21.15.46:0/903133757 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70abfff640 1 --2- 172.21.15.46:0/903133757 >> v2:172.21.15.121:3300/0 conn(0x7f70b406f210 0x7f70b4140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 -- 172.21.15.46:0/903133757 >> v2:172.21.15.121:3300/0 conn(0x7f70b406f210 msgr2=0x7f70b4140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 --2- 172.21.15.46:0/903133757 >> v2:172.21.15.121:3300/0 conn(0x7f70b406f210 0x7f70b4140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 -- 172.21.15.46:0/903133757 >> v1:172.21.15.121:6789/0 conn(0x7f70b415c2f0 legacy=0x7f70b406eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:55.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.844+0000 7f70b37fe640 1 -- 172.21.15.46:0/903133757 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b4141190 con 0x7f70b4140d40 2023-12-16T21:13:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.845+0000 7f70b37fe640 1 --2- 172.21.15.46:0/903133757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b4140d40 0x7f70b413f330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f70b40d20f0 tx=0x7f70ac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94efda3a4a0368ba server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.845+0000 7f70b27fc640 1 -- 172.21.15.46:0/903133757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac02cb20 con 0x7f70b4140d40 2023-12-16T21:13:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.845+0000 7f70b27fc640 1 -- 172.21.15.46:0/903133757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac02cc80 con 0x7f70b4140d40 2023-12-16T21:13:55.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.845+0000 7f70b27fc640 1 -- 172.21.15.46:0/903133757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0343f0 con 0x7f70b4140d40 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 -- 172.21.15.46:0/903133757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b4140d40 msgr2=0x7f70b413f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 --2- 172.21.15.46:0/903133757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b4140d40 0x7f70b413f330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f70b40d20f0 tx=0x7f70ac02bf80 comp rx=0 tx=0).stop 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 -- 172.21.15.46:0/903133757 shutdown_connections 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 --2- 172.21.15.46:0/903133757 >> v2:172.21.15.121:3300/0 conn(0x7f70b406f210 0x7f70b4140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 --2- 172.21.15.46:0/903133757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b4140d40 0x7f70b413f330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 -- 172.21.15.46:0/903133757 >> 172.21.15.46:0/903133757 conn(0x7f70b406a6b0 msgr2=0x7f70b406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 -- 172.21.15.46:0/903133757 shutdown_connections 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.846+0000 7f70b95e5640 1 -- 172.21.15.46:0/903133757 wait complete. 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.847+0000 7f70b95e5640 1 Processor -- start 2023-12-16T21:13:55.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.847+0000 7f70b95e5640 1 -- start start 2023-12-16T21:13:55.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b95e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70b406f210 0x7f70b41f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:55.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b95e5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70b4140d40 0x7f70b413c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:55.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b95e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 0x7f70b4153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:55.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b95e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b41f5290 con 0x7f70b415c2f0 2023-12-16T21:13:55.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b95e5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b41f49f0 con 0x7f70b4140d40 2023-12-16T21:13:55.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b95e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b41f4cd0 con 0x7f70b406f210 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 0x7f70b4153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 0x7f70b4153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33296/0 (socket says 172.21.15.46:33296) 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 -- 172.21.15.46:0/3624073055 learned_addr learned my addr 172.21.15.46:0/3624073055 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70abfff640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70b4140d40 0x7f70b413c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b2ffd640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70b406f210 0x7f70b41f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 -- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70b406f210 msgr2=0x7f70b41f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70b406f210 0x7f70b41f4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 -- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70b4140d40 msgr2=0x7f70b413c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70b4140d40 0x7f70b413c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 -- 172.21.15.46:0/3624073055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac009b30 con 0x7f70b415c2f0 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70abfff640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70b4140d40 0x7f70b413c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70b37fe640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 0x7f70b4153bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f70ac036040 tx=0x7f70ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.848+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac009510 con 0x7f70b415c2f0 2023-12-16T21:13:55.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.849+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac02c920 con 0x7f70b415c2f0 2023-12-16T21:13:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.849+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b41f4fb0 con 0x7f70b415c2f0 2023-12-16T21:13:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.849+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac03e690 con 0x7f70b415c2f0 2023-12-16T21:13:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.849+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b4154100 con 0x7f70b415c2f0 2023-12-16T21:13:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.850+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b4069400 con 0x7f70b415c2f0 2023-12-16T21:13:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.850+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f70ac03e7f0 con 0x7f70b415c2f0 2023-12-16T21:13:55.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.850+0000 7f70ab7fe640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f709406ae40 0x7f709406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:55.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.851+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70ac0ac100 con 0x7f70b415c2f0 2023-12-16T21:13:55.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.851+0000 7f70b2ffd640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f709406ae40 0x7f709406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:55.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.851+0000 7f70b2ffd640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f709406ae40 0x7f709406d300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f70b415cbc0 tx=0x7f70a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:55.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:55.855+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f70ac0771b0 con 0x7f70b415c2f0 2023-12-16T21:13:56.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.071+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f70b41f0d00 con 0x7f70b415c2f0 2023-12-16T21:13:56.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.071+0000 7f70ab7fe640 1 -- 172.21.15.46:0/3624073055 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f70ac07a7d0 con 0x7f70b415c2f0 2023-12-16T21:13:56.071 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:13:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f709406ae40 msgr2=0x7f709406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f709406ae40 0x7f709406d300 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f70b415cbc0 tx=0x7f70a4009040 comp rx=0 tx=0).stop 2023-12-16T21:13:56.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 msgr2=0x7f70b4153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 0x7f70b4153bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f70ac036040 tx=0x7f70ac004a40 comp rx=0 tx=0).stop 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 shutdown_connections 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f709406ae40 0x7f709406d300 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70b415c2f0 0x7f70b4153bc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70b4140d40 0x7f70b413c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 --2- 172.21.15.46:0/3624073055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70b406f210 0x7f70b41f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.073+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 >> 172.21.15.46:0/3624073055 conn(0x7f70b406a6b0 msgr2=0x7f70b415d680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.074+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 shutdown_connections 2023-12-16T21:13:56.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:56.074+0000 7f70b95e5640 1 -- 172.21.15.46:0/3624073055 wait complete. 2023-12-16T21:13:56.083 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:13:56.084 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-12-16T21:13:56.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.084 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:13:56.100 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:13:56.100 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:13:56.100 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:13:56.100 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.100 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:13:56.100 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42649sec preferred_lft 42649sec 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:13:56.101 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link tentative 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link tentative 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.103 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:13:56.169 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:13:56.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.171 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:13:56.227 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:13:56.227 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:13:56.227 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:13:56.227 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:13:56.227 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:13:56.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.228 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:56.228 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:56.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:13:56.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:13:56.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:13:56.229 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:56.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.333 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:13:56.349 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:13:56.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.350 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2023-12-16T21:13:56.405 INFO:teuthology.orchestra.run.smithi046.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-12-16T21:13:56.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.405 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:56.405 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:56.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:13:56.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-12-16T21:13:56.406 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:56.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.495 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-16T21:13:56.526 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:13:56.526 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:13:56.526 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:13:56.526 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.528 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:13:56.529 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.529 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:13:56.529 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 8e:61:01:f8:6d:cb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:13:56.529 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:13:56.529 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.529 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::8c61:1ff:fef8:6dcb/64 scope link tentative 2023-12-16T21:13:56.530 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.531 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-16T21:13:56.603 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:13:56.603 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:13:56.603 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:13:56.603 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 8e:61:01:f8:6d:cb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::8c61:1ff:fef8:6dcb/64 scope link tentative 2023-12-16T21:13:56.604 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:56.605 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-12-16T21:13:56.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-12-16T21:13:56.606 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:56.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.748 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:13:56.748 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:13:56.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.1 up 2023-12-16T21:13:56.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.1 master ceph-brx 2023-12-16T21:13:56.749 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:13:56.797 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-12-16T21:13:56.797 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:13:56.797 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-12-16T21:13:56.812 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-12-16T21:13:56.812 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:13:56.812 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-12-16T21:13:56.868 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:13:56.939 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:13:56.995 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:13:56.996 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:13:56.997 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:13:56.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:56.998 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:13:57.051 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:13:57.052 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2023-12-16T21:13:57.052 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-12-16T21:13:57.095 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:13:57.125 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:13:57.141 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:13:57.168 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:13:57.182 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:13:57.183 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:13:57.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:57.184 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:13:57.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:2023-12-16T21:13:57.190+0000 7f7705438480 -1 init, newargv = 0x561469640950 newargc=15 2023-12-16T21:13:57.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[99019]: starting ceph client 2023-12-16T21:13:57.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi046.stderr:ceph-fuse[99019]: starting fuse 2023-12-16T21:13:57.205 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:13:57.205 INFO:teuthology.orchestra.run.smithi046.stdout:44 2023-12-16T21:13:57.205 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:13:57.206 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.2")) 2023-12-16T21:13:57.207 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:13:57.330 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.2.99019.asok 2023-12-16T21:13:57.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:13:57.340 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.99019.asok 2023-12-16T21:13:57.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:57.341 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.99019.asok status 2023-12-16T21:13:57.483 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "2", 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "99019", 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 4698, 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:13:57.484 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 4698 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 334820371 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 334820371 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:13:57.485 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.4698 192.168.0.2:0/334820371", 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.2:0/334820371", 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 6, 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 21, 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:13:57.486 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:13:57.492 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:13:57.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.598+0000 7f1505f05640 1 Processor -- start 2023-12-16T21:13:57.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.598+0000 7f1505f05640 1 -- start start 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f1505f05640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f150006ea50 0x7f150006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f1505f05640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1500161d20 con 0x7f1500160fb0 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f1505f05640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f150006f370 con 0x7f15000d0ed0 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f1505f05640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f150015e760 con 0x7f150006ea50 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f14feffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1500160fb0 0x7f1500161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33440/0 (socket says 172.21.15.46:33440) 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f14feffd640 1 -- 172.21.15.46:0/3400061728 learned_addr learned my addr 172.21.15.46:0/3400061728 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f14ff7fe640 1 --2- 172.21.15.46:0/3400061728 >> v2:172.21.15.121:3300/0 conn(0x7f150006ea50 0x7f150006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428568815 0 0) 0x7f1500161d20 con 0x7f1500160fb0 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.599+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14e8003520 con 0x7f1500160fb0 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180889453 0 0) 0x7f14e8003520 con 0x7f1500160fb0 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 >> v2:172.21.15.121:3300/0 conn(0x7f150006ea50 msgr2=0x7f150006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 --2- 172.21.15.46:0/3400061728 >> v2:172.21.15.121:3300/0 conn(0x7f150006ea50 0x7f150006ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 >> v1:172.21.15.121:6789/0 conn(0x7f15000d0ed0 legacy=0x7f15000d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150015d9a0 con 0x7f1500160fb0 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f14f4001670 con 0x7f1500160fb0 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 >> v1:172.21.15.46:6789/0 conn(0x7f1500160fb0 legacy=0x7f1500161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14e8003970 0x7f14e8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14e80047c0 0x7f14e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e8005180 0x7f14e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150015e760 con 0x7f14e80047c0 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e800c120 con 0x7f14e8005180 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e800c2e0 con 0x7f14e8003970 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14feffd640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14e80047c0 0x7f14e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14ff7fe640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14e8003970 0x7f14e8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fffff640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e8005180 0x7f14e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14feffd640 1 -- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14e8003970 msgr2=0x7f14e8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14feffd640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14e8003970 0x7f14e8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14feffd640 1 -- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e8005180 msgr2=0x7f14e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14feffd640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e8005180 0x7f14e800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14fffff640 1 -- 172.21.15.46:0/3400061728 reap_dead start 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.600+0000 7f14feffd640 1 -- 172.21.15.46:0/3400061728 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e8003520 con 0x7f14e80047c0 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.601+0000 7f14feffd640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14e80047c0 0x7f14e8004c40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f14f400c240 tx=0x7f14e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241d646efae47d6e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.601+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f4005ad0 con 0x7f14e80047c0 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.601+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f4005c50 con 0x7f14e80047c0 2023-12-16T21:13:57.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.601+0000 7f14fe7fc640 1 -- 172.21.15.46:0/3400061728 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f402c8b0 con 0x7f14e80047c0 2023-12-16T21:13:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 -- 172.21.15.46:0/3400061728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14e80047c0 msgr2=0x7f14e8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14e80047c0 0x7f14e8004c40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f14f400c240 tx=0x7f14e80057f0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 -- 172.21.15.46:0/3400061728 shutdown_connections 2023-12-16T21:13:57.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 --2- 172.21.15.46:0/3400061728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14e80047c0 0x7f14e8004c40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 -- 172.21.15.46:0/3400061728 >> 172.21.15.46:0/3400061728 conn(0x7f150006a6b0 msgr2=0x7f150006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 -- 172.21.15.46:0/3400061728 shutdown_connections 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.602+0000 7f1505f05640 1 -- 172.21.15.46:0/3400061728 wait complete. 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.603+0000 7f1505f05640 1 Processor -- start 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.603+0000 7f1505f05640 1 -- start start 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f1505f05640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e80047c0 0x7f1500146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f1505f05640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15000ccef0 0x7f15000cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f1505f05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 0x7f150015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f1505f05640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15000d0a70 con 0x7f1500159e00 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f1505f05640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1500159970 con 0x7f14e80047c0 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f1505f05640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1500146850 con 0x7f15000ccef0 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 0x7f150015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 0x7f150015a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33320/0 (socket says 172.21.15.46:33320) 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 -- 172.21.15.46:0/3707281301 learned_addr learned my addr 172.21.15.46:0/3707281301 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:57.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14ff7fe640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e80047c0 0x7f1500146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14feffd640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15000ccef0 0x7f15000cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 -- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15000ccef0 msgr2=0x7f15000cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15000ccef0 0x7f15000cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 -- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e80047c0 msgr2=0x7f1500146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e80047c0 0x7f1500146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.604+0000 7f14fffff640 1 -- 172.21.15.46:0/3707281301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1500159040 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.605+0000 7f14fffff640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 0x7f150015a280 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f14f800f770 tx=0x7f14f800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.605+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f800fee0 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.605+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8006d60 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.605+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1500159320 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.605+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8003bb0 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.605+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15000cd8b0 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.606+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1500069400 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.606+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f14f8003d10 con 0x7f1500159e00 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.607+0000 7f14e7fff640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f14ec06ae70 0x7f14ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:57.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.607+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14f8088ce0 con 0x7f1500159e00 2023-12-16T21:13:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.607+0000 7f14ff7fe640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f14ec06ae70 0x7f14ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:57.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.607+0000 7f14ff7fe640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f14ec06ae70 0x7f14ec06d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009b80 tx=0x7f14f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:57.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.611+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f14f8053e20 con 0x7f1500159e00 2023-12-16T21:13:57.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.824+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1500159ab0 con 0x7f1500159e00 2023-12-16T21:13:57.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.824+0000 7f14e7fff640 1 -- 172.21.15.46:0/3707281301 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f14f801a070 con 0x7f1500159e00 2023-12-16T21:13:57.824 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:13:57.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f14ec06ae70 msgr2=0x7f14ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f14ec06ae70 0x7f14ec06d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f14f0009b80 tx=0x7f14f0008040 comp rx=0 tx=0).stop 2023-12-16T21:13:57.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 msgr2=0x7f150015a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:57.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 0x7f150015a280 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f14f800f770 tx=0x7f14f800fc30 comp rx=0 tx=0).stop 2023-12-16T21:13:57.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 shutdown_connections 2023-12-16T21:13:57.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f14ec06ae70 0x7f14ec06d330 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1500159e00 0x7f150015a280 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15000ccef0 0x7f15000cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 --2- 172.21.15.46:0/3707281301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14e80047c0 0x7f1500146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:57.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 >> 172.21.15.46:0/3707281301 conn(0x7f150006a6b0 msgr2=0x7f15000b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:57.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 shutdown_connections 2023-12-16T21:13:57.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:57.826+0000 7f1505f05640 1 -- 172.21.15.46:0/3707281301 wait complete. 2023-12-16T21:13:57.836 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:13:57.836 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:13:57.836 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2023-12-16T21:13:57.837 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-16T21:13:57.837 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T21:13:57.837 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:13:57.837 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:13:57.837 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:13:57.837 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-16T21:13:57.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:57.837 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2023-12-16T21:13:57.855 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2023-12-16T21:13:57.856 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42630sec preferred_lft 42630sec 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:57.857 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T21:13:57.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:57.858 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:13:57.858 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:13:57.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add name ceph-brx type bridge 2023-12-16T21:13:57.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr flush dev ceph-brx 2023-12-16T21:13:57.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set ceph-brx up 2023-12-16T21:13:57.859 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T21:13:57.859 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:13:58.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.036 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:13:58.064 INFO:teuthology.orchestra.run.smithi121.stdout:1 2023-12-16T21:13:58.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.066 DEBUG:teuthology.orchestra.run.smithi121:> route 2023-12-16T21:13:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2023-12-16T21:13:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:13:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-16T21:13:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-16T21:13:58.131 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:13:58.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.132 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:13:58.132 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:13:58.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-16T21:13:58.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-16T21:13:58.132 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-16T21:13:58.132 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:13:58.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.422 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2023-12-16T21:13:58.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.436 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2023-12-16T21:13:58.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.491 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:13:58.491 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:13:58.491 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:13:58.492 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-16T21:13:58.492 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:13:58.574 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-16T21:13:58.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.574 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:13:58.574 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:13:58.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-16T21:13:58.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T21:13:58.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-16T21:13:58.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-16T21:13:58.575 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-16T21:13:58.575 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:13:58.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:58.742 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:13:58.742 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:13:58.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.0 up 2023-12-16T21:13:58.742 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T21:13:58.742 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:13:58.783 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-12-16T21:13:58.784 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:13:58.784 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-16T21:13:58.842 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-16T21:13:58.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:13:58.842 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-16T21:13:58.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2023-12-16T21:13:58.963 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2023-12-16T21:13:59.019 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:13:59.020 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:13:59.021 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:13:59.021 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:13:59.021 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2023-12-16T21:13:59.021 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:13:59.021 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:13:59.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.021 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2023-12-16T21:13:59.074 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:13:59.074 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-12-16T21:13:59.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2023-12-16T21:13:59.143 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:13:59.204 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2023-12-16T21:13:59.205 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2023-12-16T21:13:59.206 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:13:59.207 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:13:59.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.207 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2023-12-16T21:13:59.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi121.stderr:2023-12-16T21:13:59.210+0000 7efdd0e3c480 -1 init, newargv = 0x5584ebc2ea10 newargc=15 2023-12-16T21:13:59.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi121.stderr:ceph-fuse[95237]: starting ceph client 2023-12-16T21:13:59.215 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi121.stderr:ceph-fuse[95237]: starting fuse 2023-12-16T21:13:59.224 INFO:teuthology.orchestra.run.smithi121.stdout:41 2023-12-16T21:13:59.225 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> import re 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> import os 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> def _find_admin_socket(client_name): 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:13:59.225 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> contents = proc_f.read() 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> if mountpoint in contents: 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> return f 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> print(_find_admin_socket("client.1")) 2023-12-16T21:13:59.226 DEBUG:teuthology.orchestra.run.smithi121:> ' 2023-12-16T21:13:59.344 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.1.95237.asok 2023-12-16T21:13:59.347 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:13:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:13:59.354 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.95237.asok 2023-12-16T21:13:59.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.354 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.95237.asok status 2023-12-16T21:13:59.490 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "1", 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "95237", 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4696, 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2023-12-16T21:13:59.491 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4696 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 849832059 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 849832059 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4696 192.168.0.1:0/849832059", 2023-12-16T21:13:59.492 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "192.168.0.1:0/849832059", 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 6, 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 21, 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklisted": false, 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2023-12-16T21:13:59.493 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-12-16T21:13:59.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:13:59.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.620+0000 7ff8b024f640 1 Processor -- start 2023-12-16T21:13:59.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.621+0000 7ff8b024f640 1 -- start start 2023-12-16T21:13:59.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8b024f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8a80d1d50 0x7ff8a806bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8b024f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a806ce00 con 0x7ff8a815d6c0 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8b024f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a806c1f0 con 0x7ff8a815d2e0 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8b024f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a806c4d0 con 0x7ff8a80d1d50 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8ae7ec640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8a80d1d50 0x7ff8a806bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8ad7ea640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff8a815d6c0 0x7ff8a806b4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33446/0 (socket says 172.21.15.46:33446) 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8ad7ea640 1 -- 172.21.15.46:0/536466003 learned_addr learned my addr 172.21.15.46:0/536466003 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:13:59.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8ae7ec640 1 -- 172.21.15.46:0/536466003 >> v1:172.21.15.121:6789/0 conn(0x7ff8a815d2e0 legacy=0x7ff8a80d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8ae7ec640 1 -- 172.21.15.46:0/536466003 >> v1:172.21.15.46:6789/0 conn(0x7ff8a815d6c0 legacy=0x7ff8a806b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.622+0000 7ff8ae7ec640 1 -- 172.21.15.46:0/536466003 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a806c7b0 con 0x7ff8a80d1d50 2023-12-16T21:13:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.623+0000 7ff8ae7ec640 1 --2- 172.21.15.46:0/536466003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a80d1d50 0x7ff8a806bbd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009d90 tx=0x7ff8a002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77714bcf80b7f03 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.623+0000 7ff8acfe9640 1 -- 172.21.15.46:0/536466003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a002cc60 con 0x7ff8a80d1d50 2023-12-16T21:13:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.623+0000 7ff8acfe9640 1 -- 172.21.15.46:0/536466003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a002cdc0 con 0x7ff8a80d1d50 2023-12-16T21:13:59.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.623+0000 7ff8acfe9640 1 -- 172.21.15.46:0/536466003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0034660 con 0x7ff8a80d1d50 2023-12-16T21:13:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 -- 172.21.15.46:0/536466003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a80d1d50 msgr2=0x7ff8a806bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 --2- 172.21.15.46:0/536466003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a80d1d50 0x7ff8a806bbd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009d90 tx=0x7ff8a002beb0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 -- 172.21.15.46:0/536466003 shutdown_connections 2023-12-16T21:13:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 --2- 172.21.15.46:0/536466003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a80d1d50 0x7ff8a806bbd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 -- 172.21.15.46:0/536466003 >> 172.21.15.46:0/536466003 conn(0x7ff8a809c580 msgr2=0x7ff8a80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:59.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 -- 172.21.15.46:0/536466003 shutdown_connections 2023-12-16T21:13:59.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.624+0000 7ff8b024f640 1 -- 172.21.15.46:0/536466003 wait complete. 2023-12-16T21:13:59.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.625+0000 7ff8b024f640 1 Processor -- start 2023-12-16T21:13:59.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.625+0000 7ff8b024f640 1 -- start start 2023-12-16T21:13:59.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8b024f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a80d1d50 0x7ff8a815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:59.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8b024f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 0x7ff8a80c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:59.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8b024f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a815d6c0 0x7ff8a813ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8b024f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a815e010 con 0x7ff8a815d2e0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8b024f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80b31b0 con 0x7ff8a80d1d50 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8b024f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80b3490 con 0x7ff8a815d6c0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 0x7ff8a80c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 0x7ff8a80c5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33346/0 (socket says 172.21.15.46:33346) 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 -- 172.21.15.46:0/3146111126 learned_addr learned my addr 172.21.15.46:0/3146111126 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8adfeb640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a80d1d50 0x7ff8a815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ae7ec640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a815d6c0 0x7ff8a813ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 -- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a815d6c0 msgr2=0x7ff8a813ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a815d6c0 0x7ff8a813ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 -- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a80d1d50 msgr2=0x7ff8a815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a80d1d50 0x7ff8a815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 -- 172.21.15.46:0/3146111126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0009a70 con 0x7ff8a815d2e0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.626+0000 7ff8ad7ea640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 0x7ff8a80c5f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff89800c3c0 tx=0x7ff89800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:59.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.627+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898005ce0 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.627+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff898005e40 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.627+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a80b3770 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.627+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8980072d0 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.627+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a80b3d30 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.628+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a80c4540 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.628+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff898007430 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.628+0000 7ff89e7fc640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff88c06ae70 0x7ff88c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.629+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff898088a50 con 0x7ff8a815d2e0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.629+0000 7ff8adfeb640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff88c06ae70 0x7ff88c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.629+0000 7ff8adfeb640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff88c06ae70 0x7ff88c06d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000f60 tx=0x7ff8a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:13:59.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.633+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff8a80c4540 con 0x7ff8a815d2e0 2023-12-16T21:13:59.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.844+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff8a80c4540 con 0x7ff8a815d2e0 2023-12-16T21:13:59.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.845+0000 7ff89e7fc640 1 -- 172.21.15.46:0/3146111126 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff8a80c4540 con 0x7ff8a815d2e0 2023-12-16T21:13:59.846 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:13:59.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff88c06ae70 msgr2=0x7ff88c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff88c06ae70 0x7ff88c06d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4000f60 tx=0x7ff8a4009040 comp rx=0 tx=0).stop 2023-12-16T21:13:59.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 msgr2=0x7ff8a80c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:13:59.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 0x7ff8a80c5f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff89800c3c0 tx=0x7ff89800c880 comp rx=0 tx=0).stop 2023-12-16T21:13:59.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 shutdown_connections 2023-12-16T21:13:59.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff88c06ae70 0x7ff88c06d330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a815d2e0 0x7ff8a80c5f20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a815d6c0 0x7ff8a813ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 --2- 172.21.15.46:0/3146111126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a80d1d50 0x7ff8a815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:13:59.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.846+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 >> 172.21.15.46:0/3146111126 conn(0x7ff8a809c580 msgr2=0x7ff8a80b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:13:59.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.847+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 shutdown_connections 2023-12-16T21:13:59.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:13:59.847+0000 7ff8b024f640 1 -- 172.21.15.46:0/3146111126 wait complete. 2023-12-16T21:13:59.856 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:13:59.857 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-12-16T21:13:59.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.857 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2023-12-16T21:13:59.873 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:13:59.873 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:13:59.873 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:13:59.873 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:59.873 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:13:59.874 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42628sec preferred_lft 42628sec 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:13:59.875 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether fa:bb:b2:32:0e:30 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::78f6:73ff:fed6:12f0/64 scope link 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether fa:bb:b2:32:0e:30 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::f8bb:b2ff:fe32:e30/64 scope link tentative 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:13:59.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.876 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:13:59.939 INFO:teuthology.orchestra.run.smithi121.stdout:1 2023-12-16T21:13:59.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.941 DEBUG:teuthology.orchestra.run.smithi121:> route 2023-12-16T21:13:59.996 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2023-12-16T21:13:59.996 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:13:59.997 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-16T21:13:59.997 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-16T21:13:59.997 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:13:59.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:13:59.997 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:13:59.997 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:13:59.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-16T21:13:59.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-16T21:13:59.997 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-16T21:13:59.997 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:14:00.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.093 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2023-12-16T21:14:00.108 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-16T21:14:00.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.109 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2023-12-16T21:14:00.163 INFO:teuthology.orchestra.run.smithi121.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-12-16T21:14:00.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.163 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:14:00.163 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:14:00.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:14:00.164 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-12-16T21:14:00.164 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:14:00.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-16T21:14:00.312 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:14:00.312 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:14:00.312 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether b2:bd:5f:9c:80:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::b0bd:5fff:fe9c:80cf/64 scope link tentative 2023-12-16T21:14:00.313 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.314 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-16T21:14:00.380 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether b2:bd:5f:9c:80:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-16T21:14:00.381 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-16T21:14:00.382 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.382 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::b0bd:5fff:fe9c:80cf/64 scope link tentative 2023-12-16T21:14:00.382 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:00.384 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-12-16T21:14:00.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.385 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:14:00.385 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:14:00.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-12-16T21:14:00.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-16T21:14:00.385 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-12-16T21:14:00.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-12-16T21:14:00.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-12-16T21:14:00.386 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:14:00.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.510 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec bash -c ' 2023-12-16T21:14:00.510 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2023-12-16T21:14:00.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.1 up 2023-12-16T21:14:00.510 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.1 master ceph-brx 2023-12-16T21:14:00.510 DEBUG:teuthology.orchestra.run.smithi121:> ') 2023-12-16T21:14:00.551 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-12-16T21:14:00.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:14:00.551 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-12-16T21:14:00.611 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-12-16T21:14:00.611 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:14:00.611 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-12-16T21:14:00.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2023-12-16T21:14:00.733 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:14:00.789 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2023-12-16T21:14:00.789 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2023-12-16T21:14:00.789 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2023-12-16T21:14:00.789 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2023-12-16T21:14:00.790 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2023-12-16T21:14:00.791 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:14:00.792 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:14:00.792 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-16T21:14:00.792 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:14:00.792 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:14:00.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.792 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2023-12-16T21:14:00.844 INFO:teuthology.orchestra.run.smithi121.stdout:41 2023-12-16T21:14:00.844 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2023-12-16T21:14:00.844 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-12-16T21:14:00.886 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2023-12-16T21:14:00.913 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2023-12-16T21:14:00.969 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2023-12-16T21:14:00.970 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:14:00.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:00.971 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2023-12-16T21:14:00.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi121.stderr:2023-12-16T21:14:00.970+0000 7f6d1820b480 -1 init, newargv = 0x55acf381d820 newargc=15 2023-12-16T21:14:00.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi121.stderr:ceph-fuse[95689]: starting ceph client 2023-12-16T21:14:00.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi121.stderr:ceph-fuse[95689]: starting fuse 2023-12-16T21:14:00.986 INFO:teuthology.orchestra.run.smithi121.stdout:41 2023-12-16T21:14:00.987 INFO:teuthology.orchestra.run.smithi121.stdout:44 2023-12-16T21:14:00.987 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> import re 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> import os 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> def _find_admin_socket(client_name): 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2023-12-16T21:14:00.987 DEBUG:teuthology.orchestra.run.smithi121:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> contents = proc_f.read() 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> if mountpoint in contents: 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> return f 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> print(_find_admin_socket("client.3")) 2023-12-16T21:14:00.988 DEBUG:teuthology.orchestra.run.smithi121:> ' 2023-12-16T21:14:01.107 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.3.95689.asok 2023-12-16T21:14:01.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-12-16T21:14:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:14:01.117 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.95689.asok 2023-12-16T21:14:01.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.95689.asok status 2023-12-16T21:14:01.253 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2023-12-16T21:14:01.253 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2023-12-16T21:14:01.253 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:14:01.253 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:14:01.253 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "3", 2023-12-16T21:14:01.253 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "95689", 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4577, 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4577 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2023-12-16T21:14:01.254 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3460108470 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: } 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.2:0", 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 3460108470 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4577 192.168.0.2:0/3460108470", 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "192.168.0.2:0/3460108470", 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 6, 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 21, 2023-12-16T21:14:01.255 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:14:01.256 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklisted": false, 2023-12-16T21:14:01.256 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2023-12-16T21:14:01.256 INFO:teuthology.orchestra.run.smithi121.stdout:} 2023-12-16T21:14:01.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.261 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:14:01.261 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:01.281 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:14:01.282 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:01.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.282 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:01.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.360 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:14:01.360 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.2 2023-12-16T21:14:01.418 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:14:01.418 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-12-16T21:14:01.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.419 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-12-16T21:14:01.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.495 DEBUG:teuthology.orchestra.run.smithi121:> stat --file-system '--printf=%T 2023-12-16T21:14:01.495 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.1 2023-12-16T21:14:01.511 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2023-12-16T21:14:01.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-12-16T21:14:01.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-16T21:14:01.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.578 DEBUG:teuthology.orchestra.run.smithi121:> stat --file-system '--printf=%T 2023-12-16T21:14:01.578 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.3 2023-12-16T21:14:01.633 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2023-12-16T21:14:01.634 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-12-16T21:14:01.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:01.634 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-12-16T21:14:01.702 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-12-16T21:14:01.738 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:01.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.855+0000 7f0abe79e640 1 Processor -- start 2023-12-16T21:14:01.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 -- start start 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ab80cded0 0x7f0ab80cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ab806b7f0 0x7f0ab806bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806c530 0x7f0ab815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab80d0500 con 0x7f0ab80cded0 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab806c9c0 con 0x7f0ab806b7f0 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.856+0000 7f0abe79e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80d0b30 con 0x7f0ab806c530 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ab80cded0 0x7f0ab80cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0ab80cded0 0x7f0ab80cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57446/0 (socket says 172.21.15.46:57446) 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 -- 172.21.15.46:0/1876788900 learned_addr learned my addr 172.21.15.46:0/1876788900 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abdf9d640 1 --2- 172.21.15.46:0/1876788900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806c530 0x7f0ab815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abcf9b640 1 --2- 172.21.15.46:0/1876788900 >> v2:172.21.15.121:3300/0 conn(0x7f0ab806b7f0 0x7f0ab806bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 -- 172.21.15.46:0/1876788900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806c530 msgr2=0x7f0ab815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:01.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 --2- 172.21.15.46:0/1876788900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806c530 0x7f0ab815c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 -- 172.21.15.46:0/1876788900 >> v2:172.21.15.121:3300/0 conn(0x7f0ab806b7f0 msgr2=0x7f0ab806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:01.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 --2- 172.21.15.46:0/1876788900 >> v2:172.21.15.121:3300/0 conn(0x7f0ab806b7f0 0x7f0ab806bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.857+0000 7f0abd79c640 1 -- 172.21.15.46:0/1876788900 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80d0220 con 0x7f0ab80cded0 2023-12-16T21:14:01.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.858+0000 7f0abd79c640 1 --2- 172.21.15.46:0/1876788900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab80cded0 0x7f0ab80cfb90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009e50 tx=0x7f0aa802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582d59d732169fd7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:01.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.858+0000 7f0ab7fff640 1 -- 172.21.15.46:0/1876788900 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa802ca70 con 0x7f0ab80cded0 2023-12-16T21:14:01.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.858+0000 7f0ab7fff640 1 -- 172.21.15.46:0/1876788900 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa802cbd0 con 0x7f0ab80cded0 2023-12-16T21:14:01.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.858+0000 7f0ab7fff640 1 -- 172.21.15.46:0/1876788900 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80343f0 con 0x7f0ab80cded0 2023-12-16T21:14:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 -- 172.21.15.46:0/1876788900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab80cded0 msgr2=0x7f0ab80cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:01.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 --2- 172.21.15.46:0/1876788900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab80cded0 0x7f0ab80cfb90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8009e50 tx=0x7f0aa802bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 -- 172.21.15.46:0/1876788900 shutdown_connections 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 --2- 172.21.15.46:0/1876788900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806c530 0x7f0ab815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 --2- 172.21.15.46:0/1876788900 >> v2:172.21.15.121:3300/0 conn(0x7f0ab806b7f0 0x7f0ab806bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 --2- 172.21.15.46:0/1876788900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab80cded0 0x7f0ab80cfb90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 -- 172.21.15.46:0/1876788900 >> 172.21.15.46:0/1876788900 conn(0x7f0ab80d0d50 msgr2=0x7f0ab80d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 -- 172.21.15.46:0/1876788900 shutdown_connections 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.859+0000 7f0abe79e640 1 -- 172.21.15.46:0/1876788900 wait complete. 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.860+0000 7f0abe79e640 1 Processor -- start 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.860+0000 7f0abe79e640 1 -- start start 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abe79e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806b7f0 0x7f0ab814f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abe79e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 0x7f0ab815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abe79e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ab80cded0 0x7f0ab80af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abe79e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab815fd20 con 0x7f0ab806c530 2023-12-16T21:14:01.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abe79e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8160000 con 0x7f0ab80cded0 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abe79e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab81602e0 con 0x7f0ab806b7f0 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 0x7f0ab815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 0x7f0ab815d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57460/0 (socket says 172.21.15.46:57460) 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 -- 172.21.15.46:0/2147661019 learned_addr learned my addr 172.21.15.46:0/2147661019 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abdf9d640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ab80cded0 0x7f0ab80af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abd79c640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806b7f0 0x7f0ab814f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 -- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806b7f0 msgr2=0x7f0ab814f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806b7f0 0x7f0ab814f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 -- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ab80cded0 msgr2=0x7f0ab80af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:01.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ab80cded0 0x7f0ab80af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abcf9b640 1 -- 172.21.15.46:0/2147661019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa8009b30 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abd79c640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806b7f0 0x7f0ab814f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.861+0000 7f0abdf9d640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ab80cded0 0x7f0ab80af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.862+0000 7f0abcf9b640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 0x7f0ab815d230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00e9b0 tx=0x7f0aac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.862+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac009670 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.862+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aac00c9a0 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.862+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab81605c0 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.862+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aac00cd50 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.862+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab815f940 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.863+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab80d1c30 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.863+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0aac013470 con 0x7f0ab806c530 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.864+0000 7f0ab5ffb640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa406ae40 0x7f0aa406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:01.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.864+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0aac089830 con 0x7f0ab806c530 2023-12-16T21:14:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.864+0000 7f0abd79c640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa406ae40 0x7f0aa406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.865+0000 7f0abd79c640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa406ae40 0x7f0aa406d300 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8035000 tx=0x7f0aa8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:01.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:01.868+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0aac0548e0 con 0x7f0ab806c530 2023-12-16T21:14:02.045 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.045+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ab8067420 con 0x7f0ab806c530 2023-12-16T21:14:02.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.046+0000 7f0ab5ffb640 1 -- 172.21.15.46:0/2147661019 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f0aac017090 con 0x7f0ab806c530 2023-12-16T21:14:02.046 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:02.046 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa406ae40 msgr2=0x7f0aa406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa406ae40 0x7f0aa406d300 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8035000 tx=0x7f0aa8030000 comp rx=0 tx=0).stop 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 msgr2=0x7f0ab815d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 0x7f0ab815d230 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0aac00e9b0 tx=0x7f0aac00ee70 comp rx=0 tx=0).stop 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 shutdown_connections 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa406ae40 0x7f0aa406d300 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ab806c530 0x7f0ab815d230 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ab80cded0 0x7f0ab80af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 --2- 172.21.15.46:0/2147661019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ab806b7f0 0x7f0ab814f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 >> 172.21.15.46:0/2147661019 conn(0x7f0ab80d0d50 msgr2=0x7f0ab80b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 shutdown_connections 2023-12-16T21:14:02.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.048+0000 7f0abe79e640 1 -- 172.21.15.46:0/2147661019 wait complete. 2023-12-16T21:14:02.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:02.061 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.170+0000 7f8fd023d640 1 Processor -- start 2023-12-16T21:14:02.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fd023d640 1 -- start start 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fd023d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc815d330 0x7f8fc80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fd023d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc80b2b10 con 0x7f8fc80d2910 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fd023d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc80d1590 con 0x7f8fc80d2d90 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fd023d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc80d1870 con 0x7f8fc815d330 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fcd7d8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8fc80d2910 0x7f8fc80d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52284/0 (socket says 172.21.15.46:52284) 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.171+0000 7f8fcd7d8640 1 -- 172.21.15.46:0/1912360468 learned_addr learned my addr 172.21.15.46:0/1912360468 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fcdfd9640 1 --2- 172.21.15.46:0/1912360468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc815d330 0x7f8fc80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839716403 0 0) 0x7f8fc80b2b10 con 0x7f8fc80d2910 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb4003520 con 0x7f8fc80d2910 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636094159 0 0) 0x7f8fc80d1590 con 0x7f8fc80d2d90 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fc80b2b10 con 0x7f8fc80d2d90 2023-12-16T21:14:02.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fcdfd9640 1 -- 172.21.15.46:0/1912360468 >> v1:172.21.15.121:6789/0 conn(0x7f8fc80d2d90 legacy=0x7f8fc80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fcdfd9640 1 -- 172.21.15.46:0/1912360468 >> v1:172.21.15.46:6789/0 conn(0x7f8fc80d2910 legacy=0x7f8fc80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.172+0000 7f8fcdfd9640 1 -- 172.21.15.46:0/1912360468 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc80d1b50 con 0x7f8fc815d330 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fcdfd9640 1 --2- 172.21.15.46:0/1912360468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc815d330 0x7f8fc80b8b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40097d0 tx=0x7f8fc402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2924d28dbc63f2e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031800 con 0x7f8fc815d330 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4031960 con 0x7f8fc815d330 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fccfd7640 1 -- 172.21.15.46:0/1912360468 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031cd0 con 0x7f8fc815d330 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fd023d640 1 -- 172.21.15.46:0/1912360468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc815d330 msgr2=0x7f8fc80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fd023d640 1 --2- 172.21.15.46:0/1912360468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc815d330 0x7f8fc80b8b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40097d0 tx=0x7f8fc402b8a0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fd023d640 1 -- 172.21.15.46:0/1912360468 shutdown_connections 2023-12-16T21:14:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fd023d640 1 --2- 172.21.15.46:0/1912360468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc815d330 0x7f8fc80b8b80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.173+0000 7f8fd023d640 1 -- 172.21.15.46:0/1912360468 >> 172.21.15.46:0/1912360468 conn(0x7f8fc8072020 msgr2=0x7f8fc8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.174+0000 7f8fd023d640 1 -- 172.21.15.46:0/1912360468 shutdown_connections 2023-12-16T21:14:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.174+0000 7f8fd023d640 1 -- 172.21.15.46:0/1912360468 wait complete. 2023-12-16T21:14:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 Processor -- start 2023-12-16T21:14:02.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 -- start start 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc80d2910 0x7f8fc81534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc80d2d90 0x7f8fc81527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 0x7f8fc815f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8152d30 con 0x7f8fc815d330 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc80c7390 con 0x7f8fc80d2910 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.175+0000 7f8fd023d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc81601f0 con 0x7f8fc80d2d90 2023-12-16T21:14:02.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 0x7f8fc815f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 0x7f8fc815f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57474/0 (socket says 172.21.15.46:57474) 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 -- 172.21.15.46:0/664363932 learned_addr learned my addr 172.21.15.46:0/664363932 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fcd7d8640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc80d2d90 0x7f8fc81527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fcdfd9640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc80d2910 0x7f8fc81534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 -- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc80d2d90 msgr2=0x7f8fc81527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc80d2d90 0x7f8fc81527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 -- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc80d2910 msgr2=0x7f8fc81534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc80d2910 0x7f8fc81534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 -- 172.21.15.46:0/664363932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009400 con 0x7f8fc815d330 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fce7da640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 0x7f8fc815f8b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000c7a0 tx=0x7f8fc000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0003b70 con 0x7f8fc815d330 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc0003cd0 con 0x7f8fc815d330 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc80c7630 con 0x7f8fc815d330 2023-12-16T21:14:02.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc000aa90 con 0x7f8fc815d330 2023-12-16T21:14:02.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.176+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc80c7bf0 con 0x7f8fc815d330 2023-12-16T21:14:02.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.177+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc80c7e10 con 0x7f8fc815d330 2023-12-16T21:14:02.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.178+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8fc000abf0 con 0x7f8fc815d330 2023-12-16T21:14:02.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.178+0000 7f8fbe7fc640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fac06ae40 0x7f8fac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.178+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fc00892a0 con 0x7f8fc815d330 2023-12-16T21:14:02.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.178+0000 7f8fcdfd9640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fac06ae40 0x7f8fac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.179+0000 7f8fcdfd9640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fac06ae40 0x7f8fac06d300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4036000 tx=0x7f8fc403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.182+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8fc0054490 con 0x7f8fc815d330 2023-12-16T21:14:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.357+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fc8069400 con 0x7f8fc815d330 2023-12-16T21:14:02.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.358+0000 7f8fbe7fc640 1 -- 172.21.15.46:0/664363932 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8fc0057ab0 con 0x7f8fc815d330 2023-12-16T21:14:02.357 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.359+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fac06ae40 msgr2=0x7f8fac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.359+0000 7f8fd023d640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fac06ae40 0x7f8fac06d300 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4036000 tx=0x7f8fc403c040 comp rx=0 tx=0).stop 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.359+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 msgr2=0x7f8fc815f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.359+0000 7f8fd023d640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 0x7f8fc815f8b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8fc000c7a0 tx=0x7f8fc000cc60 comp rx=0 tx=0).stop 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 shutdown_connections 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fac06ae40 0x7f8fac06d300 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fc815d330 0x7f8fc815f8b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fc80d2d90 0x7f8fc81527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 --2- 172.21.15.46:0/664363932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fc80d2910 0x7f8fc81534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 >> 172.21.15.46:0/664363932 conn(0x7f8fc8072020 msgr2=0x7f8fc80b6420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 shutdown_connections 2023-12-16T21:14:02.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.360+0000 7f8fd023d640 1 -- 172.21.15.46:0/664363932 wait complete. 2023-12-16T21:14:02.370 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.480+0000 7fb2101bb640 1 Processor -- start 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.480+0000 7fb2101bb640 1 -- start start 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb2101bb640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb208160fb0 0x7fb208161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb2101bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2080d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb2101bb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb208161d20 con 0x7fb20806ea50 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb2101bb640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20815e760 con 0x7fb208160fb0 2023-12-16T21:14:02.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb2101bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20815d9a0 con 0x7fb2080d0ed0 2023-12-16T21:14:02.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20e758640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2080d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb208160fb0 0x7fb208161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb208160fb0 0x7fb208161440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:47714/0 (socket says 172.21.15.46:47714) 2023-12-16T21:14:02.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20e758640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2080d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:33304/0 (socket says 172.21.15.46:33304) 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 -- 172.21.15.46:0/3442226988 learned_addr learned my addr 172.21.15.46:0/3442226988 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20df57640 1 --1- 172.21.15.46:0/3442226988 >> v1:172.21.15.121:6789/0 conn(0x7fb20806ea50 0x7fb20806ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:49484/0 (socket says 172.21.15.46:49484) 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 -- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 msgr2=0x7fb2080d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 --2- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2080d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 -- 172.21.15.46:0/3442226988 >> v1:172.21.15.121:6789/0 conn(0x7fb20806ea50 legacy=0x7fb20806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.481+0000 7fb20d756640 1 -- 172.21.15.46:0/3442226988 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20815dc80 con 0x7fb208160fb0 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.482+0000 7fb20d756640 1 --2- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb208161440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009870 tx=0x7fb1f802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea565cb2ee6a2740 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.482+0000 7fb20cf55640 1 -- 172.21.15.46:0/3442226988 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8030840 con 0x7fb208160fb0 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.482+0000 7fb20cf55640 1 -- 172.21.15.46:0/3442226988 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f80309a0 con 0x7fb208160fb0 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.482+0000 7fb20cf55640 1 -- 172.21.15.46:0/3442226988 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8030d30 con 0x7fb208160fb0 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 -- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 msgr2=0x7fb208161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 --2- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb208161440 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009870 tx=0x7fb1f802b980 comp rx=0 tx=0).stop 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 -- 172.21.15.46:0/3442226988 shutdown_connections 2023-12-16T21:14:02.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 --2- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2080d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 --2- 172.21.15.46:0/3442226988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb208161440 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 -- 172.21.15.46:0/3442226988 >> 172.21.15.46:0/3442226988 conn(0x7fb20806a6b0 msgr2=0x7fb20806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 -- 172.21.15.46:0/3442226988 shutdown_connections 2023-12-16T21:14:02.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.483+0000 7fb2101bb640 1 -- 172.21.15.46:0/3442226988 wait complete. 2023-12-16T21:14:02.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.484+0000 7fb2101bb640 1 Processor -- start 2023-12-16T21:14:02.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.484+0000 7fb2101bb640 1 -- start start 2023-12-16T21:14:02.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb2101bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 0x7fb2081e8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb2101bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2081588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb2101bb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb20813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb2101bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb208159390 con 0x7fb20806ea50 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb2101bb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb208159670 con 0x7fb208160fb0 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb2101bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb208159950 con 0x7fb2080d0ed0 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 0x7fb2081e8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 0x7fb2081e8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57488/0 (socket says 172.21.15.46:57488) 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 -- 172.21.15.46:0/1038589341 learned_addr learned my addr 172.21.15.46:0/1038589341 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20e758640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb20813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20d756640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2081588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 -- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 msgr2=0x7fb2081588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2081588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 -- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 msgr2=0x7fb20813d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb20813d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 -- 172.21.15.46:0/1038589341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f80094d0 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.485+0000 7fb20df57640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 0x7fb2081e8ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb20400c3e0 tx=0x7fb20400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.486+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2040074e0 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.486+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb204007640 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.486+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2040049e0 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.486+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb208159c30 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.486+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2080b45a0 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.487+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20815a150 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.487+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb204004b40 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.488+0000 7fb1fe7fc640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1ec06ae40 0x7fb1ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.488+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb204088fe0 con 0x7fb20806ea50 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.488+0000 7fb20d756640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1ec06ae40 0x7fb1ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.488+0000 7fb20d756640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1ec06ae40 0x7fb1ec06d300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8036000 tx=0x7fb1f8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.492+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb20815a150 con 0x7fb20806ea50 2023-12-16T21:14:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.667+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb20815a150 con 0x7fb20806ea50 2023-12-16T21:14:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.667+0000 7fb1fe7fc640 1 -- 172.21.15.46:0/1038589341 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb20815a150 con 0x7fb20806ea50 2023-12-16T21:14:02.667 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1ec06ae40 msgr2=0x7fb1ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1ec06ae40 0x7fb1ec06d300 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8036000 tx=0x7fb1f8008030 comp rx=0 tx=0).stop 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 msgr2=0x7fb2081e8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 0x7fb2081e8ef0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb20400c3e0 tx=0x7fb20400c8a0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 shutdown_connections 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1ec06ae40 0x7fb1ec06d300 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb208160fb0 0x7fb20813d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb2080d0ed0 0x7fb2081588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 --2- 172.21.15.46:0/1038589341 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb20806ea50 0x7fb2081e8ef0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 >> 172.21.15.46:0/1038589341 conn(0x7fb20806a6b0 msgr2=0x7fb20806c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 shutdown_connections 2023-12-16T21:14:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.669+0000 7fb2101bb640 1 -- 172.21.15.46:0/1038589341 wait complete. 2023-12-16T21:14:02.679 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:02.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.789+0000 7f8f82dd5640 1 Processor -- start 2023-12-16T21:14:02.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.789+0000 7f8f82dd5640 1 -- start start 2023-12-16T21:14:02.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f82dd5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 0x7f8f7c15de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f82dd5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c15a570 con 0x7f8f7c13cf70 2023-12-16T21:14:02.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f82dd5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c15b570 con 0x7f8f7c15e390 2023-12-16T21:14:02.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f82dd5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c15a7b0 con 0x7f8f7c072390 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f80b71640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8f7c13cf70 0x7f8f7c071c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52298/0 (socket says 172.21.15.46:52298) 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f80b71640 1 -- 172.21.15.46:0/2195869543 learned_addr learned my addr 172.21.15.46:0/2195869543 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7bfff640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 0x7f8f7c15de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977888606 0 0) 0x7f8f70001110 con 0x7f8f7c13cf70 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c005730 con 0x7f8f7c13cf70 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119535277 0 0) 0x7f8f7c15b570 con 0x7f8f7c15e390 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f70001110 con 0x7f8f7c15e390 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248103909 0 0) 0x7f8f6c005730 con 0x7f8f7c13cf70 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 msgr2=0x7f8f7c15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 0x7f8f7c15de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 >> v1:172.21.15.121:6789/0 conn(0x7f8f7c15e390 legacy=0x7f8f7c0cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c15aa90 con 0x7f8f7c13cf70 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8f70001800 con 0x7f8f7c13cf70 2023-12-16T21:14:02.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.790+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 >> v1:172.21.15.46:6789/0 conn(0x7f8f7c13cf70 legacy=0x7f8f7c071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7b7fe640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c005d40 0x7f8f6c006140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7b7fe640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0069d0 0x7f8f6c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7b7fe640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 0x7f8f6c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c15a7b0 con 0x7f8f6c0069d0 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c00dc40 con 0x7f8f6c005d40 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c00ddd0 con 0x7f8f7c072390 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7bfff640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0069d0 0x7f8f6c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f80b71640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c005d40 0x7f8f6c006140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f81372640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 0x7f8f6c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7bfff640 1 -- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 msgr2=0x7f8f6c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7bfff640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f7c072390 0x7f8f6c007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7bfff640 1 -- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c005d40 msgr2=0x7f8f6c006140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7bfff640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c005d40 0x7f8f6c006140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.791+0000 7f8f7bfff640 1 -- 172.21.15.46:0/2195869543 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c005730 con 0x7f8f6c0069d0 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f7bfff640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0069d0 0x7f8f6c006e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009a90 tx=0x7f8f6c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca1776b66caf3c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6802c630 con 0x7f8f6c0069d0 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6802c790 con 0x7f8f6c0069d0 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f7b7fe640 1 -- 172.21.15.46:0/2195869543 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6802cb20 con 0x7f8f6c0069d0 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f82dd5640 1 -- 172.21.15.46:0/2195869543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0069d0 msgr2=0x7f8f6c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/2195869543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f6c0069d0 0x7f8f6c006e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009a90 tx=0x7f8f6c007a00 comp rx=0 tx=0).stop 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f81372640 1 -- 172.21.15.46:0/2195869543 reap_dead start 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f82dd5640 1 -- 172.21.15.46:0/2195869543 shutdown_connections 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f82dd5640 1 -- 172.21.15.46:0/2195869543 >> 172.21.15.46:0/2195869543 conn(0x7f8f7c06c520 msgr2=0x7f8f7c06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f82dd5640 1 -- 172.21.15.46:0/2195869543 shutdown_connections 2023-12-16T21:14:02.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.792+0000 7f8f82dd5640 1 -- 172.21.15.46:0/2195869543 wait complete. 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.794+0000 7f8f82dd5640 1 Processor -- start 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.794+0000 7f8f82dd5640 1 -- start start 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.794+0000 7f8f82dd5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f6c0069d0 0x7f8f7c13be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.794+0000 7f8f82dd5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f7c0b4570 0x7f8f7c152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.794+0000 7f8f82dd5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 0x7f8f7c0b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f82dd5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c152fe0 con 0x7f8f7c152b60 2023-12-16T21:14:02.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f82dd5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c159da0 con 0x7f8f7c0b4570 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f82dd5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c159190 con 0x7f8f6c0069d0 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 0x7f8f7c0b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 0x7f8f7c0b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57502/0 (socket says 172.21.15.46:57502) 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 -- 172.21.15.46:0/603857197 learned_addr learned my addr 172.21.15.46:0/603857197 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f7bfff640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f7c0b4570 0x7f8f7c152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f80b71640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f6c0069d0 0x7f8f7c13be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 -- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f6c0069d0 msgr2=0x7f8f7c13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f6c0069d0 0x7f8f7c13be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 -- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f7c0b4570 msgr2=0x7f8f7c152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f7c0b4570 0x7f8f7c152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 -- 172.21.15.46:0/603857197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c159470 con 0x7f8f7c152b60 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f80b71640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f6c0069d0 0x7f8f7c13be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f81372640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 0x7f8f7c0b2de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c15e770 tx=0x7f8f7400fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74006d60 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74003970 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c159750 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74003ce0 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.795+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c13c920 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.796+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c069320 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.797+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8f740050d0 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.797+0000 7f8f797fa640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8f6006ae70 0x7f8f6006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.797+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f74088ba0 con 0x7f8f7c152b60 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.797+0000 7f8f80b71640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8f6006ae70 0x7f8f6006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.798+0000 7f8f80b71640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8f6006ae70 0x7f8f6006d330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8f70000c00 tx=0x7f8f70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:02.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.801+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8f74053d90 con 0x7f8f7c152b60 2023-12-16T21:14:02.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.977+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f7c159ee0 con 0x7f8f7c152b60 2023-12-16T21:14:02.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.977+0000 7f8f797fa640 1 -- 172.21.15.46:0/603857197 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8f74019020 con 0x7f8f7c152b60 2023-12-16T21:14:02.977 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8f6006ae70 msgr2=0x7f8f6006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8f6006ae70 0x7f8f6006d330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8f70000c00 tx=0x7f8f70009040 comp rx=0 tx=0).stop 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 msgr2=0x7f8f7c0b2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 0x7f8f7c0b2de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c15e770 tx=0x7f8f7400fd60 comp rx=0 tx=0).stop 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 shutdown_connections 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8f6006ae70 0x7f8f6006d330 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f7c152b60 0x7f8f7c0b2de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f7c0b4570 0x7f8f7c152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 --2- 172.21.15.46:0/603857197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f6c0069d0 0x7f8f7c13be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:02.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 >> 172.21.15.46:0/603857197 conn(0x7f8f7c06c520 msgr2=0x7f8f7c0b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:02.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 shutdown_connections 2023-12-16T21:14:02.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:02.979+0000 7f8f82dd5640 1 -- 172.21.15.46:0/603857197 wait complete. 2023-12-16T21:14:02.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 Processor -- start 2023-12-16T21:14:03.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 -- start start 2023-12-16T21:14:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0dc146460 0x7fa0dc0b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa0dc146840 0x7fa0dc06baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0dc146fd0 con 0x7fa0dc06eab0 2023-12-16T21:14:03.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc06c180 con 0x7fa0dc146460 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.097+0000 7fa0e084f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0dc06c460 con 0x7fa0dc146840 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0dc146460 0x7fa0dc0b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa0dc146460 0x7fa0dc0b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57506/0 (socket says 172.21.15.46:57506) 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 -- 172.21.15.46:0/1933970237 learned_addr learned my addr 172.21.15.46:0/1933970237 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0d1d9b640 1 --2- 172.21.15.46:0/1933970237 >> v2:172.21.15.121:3300/0 conn(0x7fa0dc146840 0x7fa0dc06baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 -- 172.21.15.46:0/1933970237 >> v2:172.21.15.121:3300/0 conn(0x7fa0dc146840 msgr2=0x7fa0dc06baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 --2- 172.21.15.46:0/1933970237 >> v2:172.21.15.121:3300/0 conn(0x7fa0dc146840 0x7fa0dc06baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 -- 172.21.15.46:0/1933970237 >> v1:172.21.15.121:6789/0 conn(0x7fa0dc06eab0 legacy=0x7fa0dc160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.098+0000 7fa0da59c640 1 -- 172.21.15.46:0/1933970237 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0dc06c740 con 0x7fa0dc146460 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0da59c640 1 --2- 172.21.15.46:0/1933970237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc146460 0x7fa0dc0b49d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0b4f40 tx=0x7fa0cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8459dfec3fba0dc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0d9d9b640 1 -- 172.21.15.46:0/1933970237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc031870 con 0x7fa0dc146460 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0d9d9b640 1 -- 172.21.15.46:0/1933970237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc0319d0 con 0x7fa0dc146460 2023-12-16T21:14:03.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0d9d9b640 1 -- 172.21.15.46:0/1933970237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc031d60 con 0x7fa0dc146460 2023-12-16T21:14:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 -- 172.21.15.46:0/1933970237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc146460 msgr2=0x7fa0dc0b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1933970237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc146460 0x7fa0dc0b49d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc0b4f40 tx=0x7fa0cc02b910 comp rx=0 tx=0).stop 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 -- 172.21.15.46:0/1933970237 shutdown_connections 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1933970237 >> v2:172.21.15.121:3300/0 conn(0x7fa0dc146840 0x7fa0dc06baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1933970237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc146460 0x7fa0dc0b49d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 -- 172.21.15.46:0/1933970237 >> 172.21.15.46:0/1933970237 conn(0x7fa0dc06a6b0 msgr2=0x7fa0dc06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.099+0000 7fa0e084f640 1 -- 172.21.15.46:0/1933970237 shutdown_connections 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.100+0000 7fa0e084f640 1 -- 172.21.15.46:0/1933970237 wait complete. 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.101+0000 7fa0e084f640 1 Processor -- start 2023-12-16T21:14:03.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.101+0000 7fa0e084f640 1 -- start start 2023-12-16T21:14:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0e084f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 0x7fa0dc0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0e084f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0dc146460 0x7fa0dc0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0e084f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0dc146840 0x7fa0dc1f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0e084f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc1f4af0 con 0x7fa0dc06eab0 2023-12-16T21:14:03.103 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0e084f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0cd9d0 con 0x7fa0dc146840 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0e084f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0dc0cdc80 con 0x7fa0dc146460 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 0x7fa0dc0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 0x7fa0dc0ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57512/0 (socket says 172.21.15.46:57512) 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 -- 172.21.15.46:0/1448429435 learned_addr learned my addr 172.21.15.46:0/1448429435 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0dad9d640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0dc146840 0x7fa0dc1f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0d1d9b640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0dc146460 0x7fa0dc0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 -- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0dc146460 msgr2=0x7fa0dc0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0dc146460 0x7fa0dc0cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 -- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0dc146840 msgr2=0x7fa0dc1f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0dc146840 0x7fa0dc1f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 -- 172.21.15.46:0/1448429435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc0094d0 con 0x7fa0dc06eab0 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0dad9d640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0dc146840 0x7fa0dc1f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:03.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0da59c640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 0x7fa0dc0ccf50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc000c00 tx=0x7fa0cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.102+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc03abe0 con 0x7fa0dc06eab0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.103+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0cc03ad40 con 0x7fa0dc06eab0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.103+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0cc044610 con 0x7fa0dc06eab0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.103+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0dc1f4150 con 0x7fa0dc06eab0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.103+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0dc1f4710 con 0x7fa0dc06eab0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.104+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0dc1f4c30 con 0x7fa0dc06eab0 2023-12-16T21:14:03.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.104+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa0cc04d450 con 0x7fa0dc06eab0 2023-12-16T21:14:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.104+0000 7fa0d37fe640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0bc06ae40 0x7fa0bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.105+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0cc02d040 con 0x7fa0dc06eab0 2023-12-16T21:14:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.105+0000 7fa0d1d9b640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0bc06ae40 0x7fa0bc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.105+0000 7fa0d1d9b640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0bc06ae40 0x7fa0bc06d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4006fd0 tx=0x7fa0c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.109+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa0dc1f4c30 con 0x7fa0dc06eab0 2023-12-16T21:14:03.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.284+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0dc1f4c30 con 0x7fa0dc06eab0 2023-12-16T21:14:03.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.285+0000 7fa0d37fe640 1 -- 172.21.15.46:0/1448429435 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fa0dc1f4c30 con 0x7fa0dc06eab0 2023-12-16T21:14:03.285 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:03.285 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0bc06ae40 msgr2=0x7fa0bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0bc06ae40 0x7fa0bc06d300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4006fd0 tx=0x7fa0c4008040 comp rx=0 tx=0).stop 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 msgr2=0x7fa0dc0ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 0x7fa0dc0ccf50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc000c00 tx=0x7fa0cc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 shutdown_connections 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0bc06ae40 0x7fa0bc06d300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0dc146840 0x7fa0dc1f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0dc146460 0x7fa0dc0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 --2- 172.21.15.46:0/1448429435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0dc06eab0 0x7fa0dc0ccf50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 >> 172.21.15.46:0/1448429435 conn(0x7fa0dc06a6b0 msgr2=0x7fa0dc06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:03.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 shutdown_connections 2023-12-16T21:14:03.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.287+0000 7fa0e084f640 1 -- 172.21.15.46:0/1448429435 wait complete. 2023-12-16T21:14:03.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:03.298 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.408+0000 7f9ea5b89640 1 Processor -- start 2023-12-16T21:14:03.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.408+0000 7f9ea5b89640 1 -- start start 2023-12-16T21:14:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.408+0000 7f9ea5b89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 0x7f9ea00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.408+0000 7f9ea5b89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea01614e0 con 0x7f9ea00ce650 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.408+0000 7f9ea5b89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea01617c0 con 0x7f9ea0160f70 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.408+0000 7f9ea5b89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0161aa0 con 0x7f9ea015d2d0 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9ea00ce650 0x7f9ea00d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52302/0 (socket says 172.21.15.46:52302) 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9ffff640 1 -- 172.21.15.46:0/3279541128 learned_addr learned my addr 172.21.15.46:0/3279541128 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9f7fe640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 0x7f9ea00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467995103 0 0) 0x7f9ea01614e0 con 0x7f9ea00ce650 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e88003520 con 0x7f9ea00ce650 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270262410 0 0) 0x7f9ea01617c0 con 0x7f9ea0160f70 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea01614e0 con 0x7f9ea0160f70 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054366982 0 0) 0x7f9e88003520 con 0x7f9ea00ce650 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 msgr2=0x7f9ea00b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 0x7f9ea00b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 >> v1:172.21.15.121:6789/0 conn(0x7f9ea0160f70 legacy=0x7f9ea00cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea0161d80 con 0x7f9ea00ce650 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f9e980019f0 con 0x7f9ea00ce650 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 >> v1:172.21.15.46:6789/0 conn(0x7f9ea00ce650 legacy=0x7f9ea00d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e88003b30 0x7f9e88003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 0x7f9e880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d00 0x7f9e88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0161aa0 con 0x7f9e88004d00 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8800ba30 con 0x7f9e88003b30 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8800bba0 con 0x7f9ea015d2d0 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9ffff640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d00 0x7f9e88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9f7fe640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e88003b30 0x7f9e88003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.409+0000 7f9e9effd640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 0x7f9e880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9ffff640 1 -- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 msgr2=0x7f9e880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9ffff640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea015d2d0 0x7f9e880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9ffff640 1 -- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e88003b30 msgr2=0x7f9e88003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:03.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9ffff640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e88003b30 0x7f9e88003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9ffff640 1 -- 172.21.15.46:0/3279541128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e88003520 con 0x7f9e88004d00 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9ffff640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d00 0x7f9e88005180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800a7a0 tx=0x7f9e880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b9d9b39a2218ff server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e98034630 con 0x7f9e88004d00 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9802f070 con 0x7f9e88004d00 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.410+0000 7f9e9e7fc640 1 -- 172.21.15.46:0/3279541128 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e980349f0 con 0x7f9e88004d00 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9ea5b89640 1 -- 172.21.15.46:0/3279541128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d00 msgr2=0x7f9e88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/3279541128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e88004d00 0x7f9e88005180 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9e9800a7a0 tx=0x7f9e880057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9e9ffff640 1 -- 172.21.15.46:0/3279541128 reap_dead start 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9ea5b89640 1 -- 172.21.15.46:0/3279541128 shutdown_connections 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9ea5b89640 1 -- 172.21.15.46:0/3279541128 >> 172.21.15.46:0/3279541128 conn(0x7f9ea0072230 msgr2=0x7f9ea00cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9ea5b89640 1 -- 172.21.15.46:0/3279541128 shutdown_connections 2023-12-16T21:14:03.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.411+0000 7f9ea5b89640 1 -- 172.21.15.46:0/3279541128 wait complete. 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.412+0000 7f9ea5b89640 1 Processor -- start 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.412+0000 7f9ea5b89640 1 -- start start 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9ea5b89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 0x7f9ea015f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9ea5b89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ea00b51c0 0x7f9ea014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9ea5b89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea014e6f0 0x7f9ea00cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9ea5b89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00cd520 con 0x7f9ea00b4de0 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9ea5b89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea015f950 con 0x7f9ea00b51c0 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9ea5b89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea014eb70 con 0x7f9ea014e6f0 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 0x7f9ea015f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.416 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 0x7f9ea015f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57528/0 (socket says 172.21.15.46:57528) 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 -- 172.21.15.46:0/1351755346 learned_addr learned my addr 172.21.15.46:0/1351755346 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 -- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea014e6f0 msgr2=0x7f9ea00cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea014e6f0 0x7f9ea00cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 -- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ea00b51c0 msgr2=0x7f9ea014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ea00b51c0 0x7f9ea014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.413+0000 7f9e9f7fe640 1 -- 172.21.15.46:0/1351755346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea00ca670 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.414+0000 7f9e9f7fe640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 0x7f9ea015f410 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9e900119a0 tx=0x7f9e90011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.414+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000fa10 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.414+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9000fb70 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.414+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea00ca950 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.414+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9000fee0 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.414+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea00caf10 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.415+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea0069400 con 0x7f9ea00b4de0 2023-12-16T21:14:03.417 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.415+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9e90014440 con 0x7f9ea00b4de0 2023-12-16T21:14:03.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.416+0000 7f9e87fff640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9e8c06ae70 0x7f9e8c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.416+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e90089df0 con 0x7f9ea00b4de0 2023-12-16T21:14:03.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.416+0000 7f9e9effd640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9e8c06ae70 0x7f9e8c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.416+0000 7f9e9effd640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9e8c06ae70 0x7f9e8c06d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00cd700 tx=0x7f9e94009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.420+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9e90054ea0 con 0x7f9ea00b4de0 2023-12-16T21:14:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.594+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ea00d0aa0 con 0x7f9ea00b4de0 2023-12-16T21:14:03.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.594+0000 7f9e87fff640 1 -- 172.21.15.46:0/1351755346 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e900584c0 con 0x7f9ea00b4de0 2023-12-16T21:14:03.594 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9e8c06ae70 msgr2=0x7f9e8c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9e8c06ae70 0x7f9e8c06d330 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9ea00cd700 tx=0x7f9e94009040 comp rx=0 tx=0).stop 2023-12-16T21:14:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 msgr2=0x7f9ea015f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 0x7f9ea015f410 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9e900119a0 tx=0x7f9e90011e60 comp rx=0 tx=0).stop 2023-12-16T21:14:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 shutdown_connections 2023-12-16T21:14:03.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9e8c06ae70 0x7f9e8c06d330 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ea014e6f0 0x7f9ea00cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ea00b51c0 0x7f9ea014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 --2- 172.21.15.46:0/1351755346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ea00b4de0 0x7f9ea015f410 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 >> 172.21.15.46:0/1351755346 conn(0x7f9ea0072230 msgr2=0x7f9ea00c82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.596+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 shutdown_connections 2023-12-16T21:14:03.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.597+0000 7f9ea5b89640 1 -- 172.21.15.46:0/1351755346 wait complete. 2023-12-16T21:14:03.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.717+0000 7fa86f333640 1 Processor -- start 2023-12-16T21:14:03.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.717+0000 7fa86f333640 1 -- start start 2023-12-16T21:14:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86f333640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa86815e730 0x7fa86815abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86f333640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86815e2a0 con 0x7fa868143490 2023-12-16T21:14:03.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86f333640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86815b240 con 0x7fa8680b1400 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86f333640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86815b520 con 0x7fa86815e730 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86d8d0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa86815e730 0x7fa86815abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86d0cf640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa868143490 0x7fa8680b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52306/0 (socket says 172.21.15.46:52306) 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86d0cf640 1 -- 172.21.15.46:0/1490501550 learned_addr learned my addr 172.21.15.46:0/1490501550 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86d8d0640 1 -- 172.21.15.46:0/1490501550 >> v1:172.21.15.121:6789/0 conn(0x7fa8680b1400 legacy=0x7fa86815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86d8d0640 1 -- 172.21.15.46:0/1490501550 >> v1:172.21.15.46:6789/0 conn(0x7fa868143490 legacy=0x7fa8680b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa86d8d0640 1 -- 172.21.15.46:0/1490501550 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86815dc50 con 0x7fa86815e730 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.718+0000 7fa85ffff640 1 -- 172.21.15.46:0/1490501550 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715399297 0 0) 0x7fa86815e2a0 con 0x7fa868143490 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.719+0000 7fa86d8d0640 1 --2- 172.21.15.46:0/1490501550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86815e730 0x7fa86815abf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa860009d90 tx=0x7fa86002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f91c8c9f9e6bd7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.719+0000 7fa85ffff640 1 -- 172.21.15.46:0/1490501550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86002cd00 con 0x7fa86815e730 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.719+0000 7fa85ffff640 1 -- 172.21.15.46:0/1490501550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86002ce60 con 0x7fa86815e730 2023-12-16T21:14:03.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.719+0000 7fa85ffff640 1 -- 172.21.15.46:0/1490501550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa860034660 con 0x7fa86815e730 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 -- 172.21.15.46:0/1490501550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86815e730 msgr2=0x7fa86815abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 --2- 172.21.15.46:0/1490501550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86815e730 0x7fa86815abf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa860009d90 tx=0x7fa86002bf50 comp rx=0 tx=0).stop 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 -- 172.21.15.46:0/1490501550 shutdown_connections 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 --2- 172.21.15.46:0/1490501550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa86815e730 0x7fa86815abf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 -- 172.21.15.46:0/1490501550 >> 172.21.15.46:0/1490501550 conn(0x7fa86806a4d0 msgr2=0x7fa86806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 -- 172.21.15.46:0/1490501550 shutdown_connections 2023-12-16T21:14:03.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.720+0000 7fa86f333640 1 -- 172.21.15.46:0/1490501550 wait complete. 2023-12-16T21:14:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.721+0000 7fa86f333640 1 Processor -- start 2023-12-16T21:14:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.721+0000 7fa86f333640 1 -- start start 2023-12-16T21:14:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86f333640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 0x7fa8681f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86f333640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa868143490 0x7fa86813adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86f333640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86815e730 0x7fa8680aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86f333640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8681f6690 con 0x7fa8680b1400 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86f333640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa868159b50 con 0x7fa86815e730 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86f333640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8681592b0 con 0x7fa868143490 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 0x7fa8681f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 0x7fa8681f6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57540/0 (socket says 172.21.15.46:57540) 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 -- 172.21.15.46:0/3501921296 learned_addr learned my addr 172.21.15.46:0/3501921296 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 -- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa868143490 msgr2=0x7fa86813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa868143490 0x7fa86813adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 -- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86815e730 msgr2=0x7fa8680aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86815e730 0x7fa8680aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 -- 172.21.15.46:0/3501921296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa860009a70 con 0x7fa8680b1400 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.722+0000 7fa86d0cf640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 0x7fa8681f6150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa86400c4d0 tx=0x7fa86400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.723+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640074e0 con 0x7fa8680b1400 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.723+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864007640 con 0x7fa8680b1400 2023-12-16T21:14:03.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.723+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa864004a20 con 0x7fa8680b1400 2023-12-16T21:14:03.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.723+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa868159590 con 0x7fa8680b1400 2023-12-16T21:14:03.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.723+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8680aca10 con 0x7fa8680b1400 2023-12-16T21:14:03.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.724+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa868069400 con 0x7fa8680b1400 2023-12-16T21:14:03.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.724+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa864004b80 con 0x7fa8680b1400 2023-12-16T21:14:03.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.725+0000 7fa85dffb640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa84c06ae40 0x7fa84c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:03.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.725+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8640888a0 con 0x7fa8680b1400 2023-12-16T21:14:03.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.725+0000 7fa86c8ce640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa84c06ae40 0x7fa84c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:03.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.725+0000 7fa86c8ce640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa84c06ae40 0x7fa84c06d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa858000f60 tx=0x7fa8580079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:03.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.729+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa864053940 con 0x7fa8680b1400 2023-12-16T21:14:03.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.904+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8680b2820 con 0x7fa8680b1400 2023-12-16T21:14:03.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.904+0000 7fa85dffb640 1 -- 172.21.15.46:0/3501921296 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa864018020 con 0x7fa8680b1400 2023-12-16T21:14:03.904 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa84c06ae40 msgr2=0x7fa84c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa84c06ae40 0x7fa84c06d300 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa858000f60 tx=0x7fa8580079b0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 msgr2=0x7fa8681f6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 0x7fa8681f6150 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa86400c4d0 tx=0x7fa86400c990 comp rx=0 tx=0).stop 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 shutdown_connections 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa84c06ae40 0x7fa84c06d300 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86815e730 0x7fa8680aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa868143490 0x7fa86813adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 --2- 172.21.15.46:0/3501921296 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8680b1400 0x7fa8681f6150 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:03.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 >> 172.21.15.46:0/3501921296 conn(0x7fa86806a4d0 msgr2=0x7fa86815a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:03.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 shutdown_connections 2023-12-16T21:14:03.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:03.906+0000 7fa86f333640 1 -- 172.21.15.46:0/3501921296 wait complete. 2023-12-16T21:14:03.917 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.026+0000 7fbf6bbc2640 1 Processor -- start 2023-12-16T21:14:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.027+0000 7fbf6bbc2640 1 -- start start 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.027+0000 7fbf6bbc2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf640b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.027+0000 7fbf6bbc2640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf640b2b10 con 0x7fbf640d2910 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.027+0000 7fbf6bbc2640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf640d1590 con 0x7fbf6415d330 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.027+0000 7fbf6bbc2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640d1870 con 0x7fbf640d3900 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6915d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbf640d2910 0x7fbf640d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52310/0 (socket says 172.21.15.46:52310) 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6915d640 1 -- 172.21.15.46:0/1316111975 learned_addr learned my addr 172.21.15.46:0/1316111975 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf640b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429031503 0 0) 0x7fbf640b2b10 con 0x7fbf640d2910 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf50003520 con 0x7fbf640d2910 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2454068136 0 0) 0x7fbf50003520 con 0x7fbf640d2910 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 msgr2=0x7fbf640b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf640b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 >> v1:172.21.15.121:6789/0 conn(0x7fbf6415d330 legacy=0x7fbf640b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf640d1b50 con 0x7fbf640d2910 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fbf54001670 con 0x7fbf640d2910 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf640b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 >> v1:172.21.15.46:6789/0 conn(0x7fbf640d2910 legacy=0x7fbf640d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf500039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50003ee0 0x7fbf500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf50004d00 0x7fbf50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640d1870 con 0x7fbf50004d00 2023-12-16T21:14:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5000ba90 con 0x7fbf50003ee0 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5000bc50 con 0x7fbf640d3900 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf50004d00 0x7fbf50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6995e640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf500039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.028+0000 7fbf6915d640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50003ee0 0x7fbf500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6a15f640 1 -- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 msgr2=0x7fbf500039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf640d3900 0x7fbf500039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6a15f640 1 -- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50003ee0 msgr2=0x7fbf500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50003ee0 0x7fbf500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6a15f640 1 -- 172.21.15.46:0/1316111975 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c009a70 con 0x7fbf50004d00 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf50004d00 0x7fbf50005180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00e350 tx=0x7fbf500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d765d1f25de13508 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c02cd70 con 0x7fbf50004d00 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c00f040 con 0x7fbf50004d00 2023-12-16T21:14:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.029+0000 7fbf6895c640 1 -- 172.21.15.46:0/1316111975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c003bb0 con 0x7fbf50004d00 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/1316111975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf50004d00 msgr2=0x7fbf50005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/1316111975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf50004d00 0x7fbf50005180 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c00e350 tx=0x7fbf500057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6a15f640 1 -- 172.21.15.46:0/1316111975 reap_dead start 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/1316111975 shutdown_connections 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/1316111975 >> 172.21.15.46:0/1316111975 conn(0x7fbf64072020 msgr2=0x7fbf64072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/1316111975 shutdown_connections 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.030+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/1316111975 wait complete. 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.031+0000 7fbf6bbc2640 1 Processor -- start 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.031+0000 7fbf6bbc2640 1 -- start start 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6bbc2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50004d00 0x7fbf641536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6bbc2640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf64152a90 0x7fbf640c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6bbc2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 0x7fbf640c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6bbc2640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640cc070 con 0x7fbf64152540 2023-12-16T21:14:04.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6bbc2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640cc350 con 0x7fbf50004d00 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6bbc2640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf640cc630 con 0x7fbf64152a90 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6a15f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 0x7fbf640c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6a15f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 0x7fbf640c6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57566/0 (socket says 172.21.15.46:57566) 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6a15f640 1 -- 172.21.15.46:0/257375576 learned_addr learned my addr 172.21.15.46:0/257375576 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6915d640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf64152a90 0x7fbf640c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.032+0000 7fbf6995e640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50004d00 0x7fbf641536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6a15f640 1 -- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf64152a90 msgr2=0x7fbf640c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf64152a90 0x7fbf640c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6a15f640 1 -- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50004d00 msgr2=0x7fbf641536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50004d00 0x7fbf641536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6a15f640 1 -- 172.21.15.46:0/257375576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf640cc910 con 0x7fbf64152540 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6a15f640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 0x7fbf640c6520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c036040 tx=0x7fbf5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c004240 con 0x7fbf64152540 2023-12-16T21:14:04.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5c0043a0 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf640ccbf0 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5c004730 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.033+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf64153bf0 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.034+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf64153de0 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.034+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbf5c03f430 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.035+0000 7fbf5a7fc640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbf4806ae40 0x7fbf4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.035+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf5c0ac3d0 con 0x7fbf64152540 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.035+0000 7fbf6995e640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbf4806ae40 0x7fbf4806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.035+0000 7fbf6995e640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbf4806ae40 0x7fbf4806d300 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf60003700 tx=0x7fbf60009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.039+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fbf5c077510 con 0x7fbf64152540 2023-12-16T21:14:04.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.216+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf640c74c0 con 0x7fbf64152540 2023-12-16T21:14:04.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.216+0000 7fbf5a7fc640 1 -- 172.21.15.46:0/257375576 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf640c74c0 con 0x7fbf64152540 2023-12-16T21:14:04.216 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:04.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbf4806ae40 msgr2=0x7fbf4806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbf4806ae40 0x7fbf4806d300 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf60003700 tx=0x7fbf60009040 comp rx=0 tx=0).stop 2023-12-16T21:14:04.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 msgr2=0x7fbf640c6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 0x7fbf640c6520 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c036040 tx=0x7fbf5c009750 comp rx=0 tx=0).stop 2023-12-16T21:14:04.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 shutdown_connections 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbf4806ae40 0x7fbf4806d300 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbf64152540 0x7fbf640c6520 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf64152a90 0x7fbf640c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 --2- 172.21.15.46:0/257375576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf50004d00 0x7fbf641536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 >> 172.21.15.46:0/257375576 conn(0x7fbf64072020 msgr2=0x7fbf641602e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 shutdown_connections 2023-12-16T21:14:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.218+0000 7fbf6bbc2640 1 -- 172.21.15.46:0/257375576 wait complete. 2023-12-16T21:14:04.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.337+0000 7f831f961640 1 Processor -- start 2023-12-16T21:14:04.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.337+0000 7f831f961640 1 -- start start 2023-12-16T21:14:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831f961640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f83180b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831f961640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8318160f70 0x7f83180cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831f961640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83181614e0 con 0x7f83180ce480 2023-12-16T21:14:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831f961640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83181617c0 con 0x7f8318160f70 2023-12-16T21:14:04.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831f961640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8318161aa0 con 0x7f831815d2d0 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f83180b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831cefc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8318160f70 0x7f83180cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f83180b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:33452/0 (socket says 172.21.15.46:33452) 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 -- 172.21.15.46:0/549606301 learned_addr learned my addr 172.21.15.46:0/549606301 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f830ffff640 1 -- 172.21.15.46:0/549606301 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981520178 0 0) 0x7f83181614e0 con 0x7f83180ce480 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f830ffff640 1 -- 172.21.15.46:0/549606301 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8304003520 con 0x7f83180ce480 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 -- 172.21.15.46:0/549606301 >> v2:172.21.15.121:3300/0 conn(0x7f8318160f70 msgr2=0x7f83180cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 --2- 172.21.15.46:0/549606301 >> v2:172.21.15.121:3300/0 conn(0x7f8318160f70 0x7f83180cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 -- 172.21.15.46:0/549606301 >> v1:172.21.15.121:6789/0 conn(0x7f83180ce480 legacy=0x7f831806b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831d6fd640 1 -- 172.21.15.46:0/549606301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318161d80 con 0x7f831815d2d0 2023-12-16T21:14:04.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.338+0000 7f831cefc640 1 --2- 172.21.15.46:0/549606301 >> v2:172.21.15.121:3300/0 conn(0x7f8318160f70 0x7f83180cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.339+0000 7f831d6fd640 1 --2- 172.21.15.46:0/549606301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f83180b8b90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8314009870 tx=0x7f831402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb270c9b14c0465 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.339+0000 7f830ffff640 1 -- 172.21.15.46:0/549606301 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314030840 con 0x7f831815d2d0 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.339+0000 7f830ffff640 1 -- 172.21.15.46:0/549606301 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83140309a0 con 0x7f831815d2d0 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.339+0000 7f830ffff640 1 -- 172.21.15.46:0/549606301 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314030d30 con 0x7f831815d2d0 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 -- 172.21.15.46:0/549606301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 msgr2=0x7f83180b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 --2- 172.21.15.46:0/549606301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f83180b8b90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8314009870 tx=0x7f831402b980 comp rx=0 tx=0).stop 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 -- 172.21.15.46:0/549606301 shutdown_connections 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 --2- 172.21.15.46:0/549606301 >> v2:172.21.15.121:3300/0 conn(0x7f8318160f70 0x7f83180cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 --2- 172.21.15.46:0/549606301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f83180b8b90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 -- 172.21.15.46:0/549606301 >> 172.21.15.46:0/549606301 conn(0x7f8318072230 msgr2=0x7f83180cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 -- 172.21.15.46:0/549606301 shutdown_connections 2023-12-16T21:14:04.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.340+0000 7f831f961640 1 -- 172.21.15.46:0/549606301 wait complete. 2023-12-16T21:14:04.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.341+0000 7f831f961640 1 Processor -- start 2023-12-16T21:14:04.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.341+0000 7f831f961640 1 -- start start 2023-12-16T21:14:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831f961640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83180ce480 0x7f8318157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831f961640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f831815af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831f961640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 0x7f83180cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831f961640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831815da30 con 0x7f8318160f70 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831f961640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831815dcd0 con 0x7f83180ce480 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831f961640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831815b970 con 0x7f831815d2d0 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 0x7f83180cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 0x7f83180cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57574/0 (socket says 172.21.15.46:57574) 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 -- 172.21.15.46:0/2844677585 learned_addr learned my addr 172.21.15.46:0/2844677585 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831cefc640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f831815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831d6fd640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83180ce480 0x7f8318157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 -- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 msgr2=0x7f831815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f831815af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 -- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83180ce480 msgr2=0x7f8318157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83180ce480 0x7f8318157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831defe640 1 -- 172.21.15.46:0/2844677585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8308009400 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831cefc640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f831815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.342+0000 7f831d6fd640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83180ce480 0x7f8318157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.343+0000 7f831defe640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 0x7f83180cc2e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f831000c790 tx=0x7f831000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.343+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831000cea0 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.343+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8310003b70 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.343+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83140094d0 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.343+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831000a850 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.343+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831815bf30 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.344+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8318069400 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.344+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f831000a9b0 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.345+0000 7f830dffb640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82fc06ae40 0x7f82fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.345+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8310089560 con 0x7f8318160f70 2023-12-16T21:14:04.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.345+0000 7f831d6fd640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82fc06ae40 0x7f82fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.345+0000 7f831d6fd640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82fc06ae40 0x7f82fc06d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8314036000 tx=0x7f8314008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.349+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f83100580a0 con 0x7f8318160f70 2023-12-16T21:14:04.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.526+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f831815de10 con 0x7f8318160f70 2023-12-16T21:14:04.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.527+0000 7f830dffb640 1 -- 172.21.15.46:0/2844677585 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f8310057c30 con 0x7f8318160f70 2023-12-16T21:14:04.527 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:04.527 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82fc06ae40 msgr2=0x7f82fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82fc06ae40 0x7f82fc06d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8314036000 tx=0x7f8314008030 comp rx=0 tx=0).stop 2023-12-16T21:14:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 msgr2=0x7f83180cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 0x7f83180cc2e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f831000c790 tx=0x7f831000cc50 comp rx=0 tx=0).stop 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 shutdown_connections 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82fc06ae40 0x7f82fc06d300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8318160f70 0x7f83180cc2e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f831815d2d0 0x7f831815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 --2- 172.21.15.46:0/2844677585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83180ce480 0x7f8318157dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.529+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 >> 172.21.15.46:0/2844677585 conn(0x7f8318072230 msgr2=0x7f83180c6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.530+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 shutdown_connections 2023-12-16T21:14:04.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.530+0000 7f831f961640 1 -- 172.21.15.46:0/2844677585 wait complete. 2023-12-16T21:14:04.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:04.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.651+0000 7fe48aab4640 1 Processor -- start 2023-12-16T21:14:04.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.651+0000 7fe48aab4640 1 -- start start 2023-12-16T21:14:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe48aab4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe48406ea50 0x7fe48406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe48aab4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe484160fb0 0x7fe484161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe48aab4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe484161d20 con 0x7fe4840b3ff0 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe48aab4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4840d1ee0 con 0x7fe484160fb0 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe48aab4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48406f370 con 0x7fe48406ea50 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe483fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe484160fb0 0x7fe484161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe489051640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe4840b3ff0 0x7fe4840b4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52314/0 (socket says 172.21.15.46:52314) 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe489051640 1 -- 172.21.15.46:0/1134820643 learned_addr learned my addr 172.21.15.46:0/1134820643 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe488850640 1 --2- 172.21.15.46:0/1134820643 >> v2:172.21.15.121:3300/0 conn(0x7fe48406ea50 0x7fe48406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.652+0000 7fe4837fe640 1 -- 172.21.15.46:0/1134820643 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159633851 0 0) 0x7fe484161d20 con 0x7fe4840b3ff0 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe4837fe640 1 -- 172.21.15.46:0/1134820643 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe46c003520 con 0x7fe4840b3ff0 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe483fff640 1 -- 172.21.15.46:0/1134820643 >> v2:172.21.15.121:3300/0 conn(0x7fe48406ea50 msgr2=0x7fe48406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe483fff640 1 --2- 172.21.15.46:0/1134820643 >> v2:172.21.15.121:3300/0 conn(0x7fe48406ea50 0x7fe48406ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe483fff640 1 -- 172.21.15.46:0/1134820643 >> v1:172.21.15.46:6789/0 conn(0x7fe4840b3ff0 legacy=0x7fe4840b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.653 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe483fff640 1 -- 172.21.15.46:0/1134820643 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48415e760 con 0x7fe484160fb0 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.653+0000 7fe483fff640 1 --2- 172.21.15.46:0/1134820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe484160fb0 0x7fe484161440 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe48406f680 tx=0x7fe47802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9424ecc9b80cb8b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe4837fe640 1 -- 172.21.15.46:0/1134820643 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47802e070 con 0x7fe484160fb0 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe4837fe640 1 -- 172.21.15.46:0/1134820643 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4780315a0 con 0x7fe484160fb0 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe4837fe640 1 -- 172.21.15.46:0/1134820643 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478031930 con 0x7fe484160fb0 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 -- 172.21.15.46:0/1134820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe484160fb0 msgr2=0x7fe484161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 --2- 172.21.15.46:0/1134820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe484160fb0 0x7fe484161440 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe48406f680 tx=0x7fe47802b840 comp rx=0 tx=0).stop 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 -- 172.21.15.46:0/1134820643 shutdown_connections 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 --2- 172.21.15.46:0/1134820643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe484160fb0 0x7fe484161440 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 --2- 172.21.15.46:0/1134820643 >> v2:172.21.15.121:3300/0 conn(0x7fe48406ea50 0x7fe48406ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 -- 172.21.15.46:0/1134820643 >> 172.21.15.46:0/1134820643 conn(0x7fe48406a6b0 msgr2=0x7fe48406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 -- 172.21.15.46:0/1134820643 shutdown_connections 2023-12-16T21:14:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.654+0000 7fe48aab4640 1 -- 172.21.15.46:0/1134820643 wait complete. 2023-12-16T21:14:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.655+0000 7fe48aab4640 1 Processor -- start 2023-12-16T21:14:04.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.655+0000 7fe48aab4640 1 -- start start 2023-12-16T21:14:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe48aab4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 0x7fe48413adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe48aab4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4840b3ff0 0x7fe4841547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe48aab4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe484160fb0 0x7fe484154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe48aab4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48413c3f0 con 0x7fe48406ea50 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe48aab4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48413c6d0 con 0x7fe484160fb0 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe48aab4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48413c9b0 con 0x7fe4840b3ff0 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe488850640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 0x7fe48413adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe488850640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 0x7fe48413adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57592/0 (socket says 172.21.15.46:57592) 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.656+0000 7fe488850640 1 -- 172.21.15.46:0/164022951 learned_addr learned my addr 172.21.15.46:0/164022951 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe489051640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe484160fb0 0x7fe484154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe483fff640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4840b3ff0 0x7fe4841547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe488850640 1 -- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4840b3ff0 msgr2=0x7fe4841547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe488850640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4840b3ff0 0x7fe4841547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe488850640 1 -- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe484160fb0 msgr2=0x7fe484154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe488850640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe484160fb0 0x7fe484154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe488850640 1 -- 172.21.15.46:0/164022951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe478009400 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe488850640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 0x7fe48413adb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007e40 tx=0x7fe47c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c014620 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c00ee90 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48413cc90 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0149c0 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.657+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48413b2f0 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.658+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe484069400 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.659+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe47c014b20 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.659+0000 7fe4817fa640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe47006ae40 0x7fe47006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.659+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe47c089f60 con 0x7fe48406ea50 2023-12-16T21:14:04.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.659+0000 7fe483fff640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe47006ae40 0x7fe47006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.660+0000 7fe483fff640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe47006ae40 0x7fe47006d300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe478036000 tx=0x7fe47803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.663+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe47c055010 con 0x7fe48406ea50 2023-12-16T21:14:04.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.842+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe484070d30 con 0x7fe48406ea50 2023-12-16T21:14:04.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.842+0000 7fe4817fa640 1 -- 172.21.15.46:0/164022951 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe47c058630 con 0x7fe48406ea50 2023-12-16T21:14:04.842 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:04.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.844+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe47006ae40 msgr2=0x7fe47006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.844+0000 7fe48aab4640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe47006ae40 0x7fe47006d300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe478036000 tx=0x7fe47803d040 comp rx=0 tx=0).stop 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.844+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 msgr2=0x7fe48413adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.844+0000 7fe48aab4640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 0x7fe48413adb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe47c007e40 tx=0x7fe47c00e4b0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 shutdown_connections 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe47006ae40 0x7fe47006d300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe484160fb0 0x7fe484154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4840b3ff0 0x7fe4841547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 --2- 172.21.15.46:0/164022951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe48406ea50 0x7fe48413adb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 >> 172.21.15.46:0/164022951 conn(0x7fe48406a6b0 msgr2=0x7fe484157390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 shutdown_connections 2023-12-16T21:14:04.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.845+0000 7fe48aab4640 1 -- 172.21.15.46:0/164022951 wait complete. 2023-12-16T21:14:04.855 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:04.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.965+0000 7f3afa415640 1 Processor -- start 2023-12-16T21:14:04.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.965+0000 7f3afa415640 1 -- start start 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3afa415640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3af415d330 0x7f3af40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3afa415640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40b2b10 con 0x7f3af40d2910 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3afa415640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40d1590 con 0x7f3af40d3900 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3afa415640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40d1870 con 0x7f3af415d330 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3af3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3af415d330 0x7f3af40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3af37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3af40d2910 0x7f3af40d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52316/0 (socket says 172.21.15.46:52316) 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3af37fe640 1 -- 172.21.15.46:0/205256741 learned_addr learned my addr 172.21.15.46:0/205256741 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3af3fff640 1 -- 172.21.15.46:0/205256741 >> v1:172.21.15.121:6789/0 conn(0x7f3af40d3900 legacy=0x7f3af40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3af3fff640 1 -- 172.21.15.46:0/205256741 >> v1:172.21.15.46:6789/0 conn(0x7f3af40d2910 legacy=0x7f3af40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.966+0000 7f3af3fff640 1 -- 172.21.15.46:0/205256741 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40d1b50 con 0x7f3af415d330 2023-12-16T21:14:04.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.967+0000 7f3af3fff640 1 --2- 172.21.15.46:0/205256741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af415d330 0x7f3af40b8b80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097a0 tx=0x7f3ae402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb443aca0bd129ab server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.967+0000 7f3af2ffd640 1 -- 172.21.15.46:0/205256741 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae402e070 con 0x7f3af415d330 2023-12-16T21:14:04.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.967+0000 7f3af2ffd640 1 -- 172.21.15.46:0/205256741 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae40315a0 con 0x7f3af415d330 2023-12-16T21:14:04.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.967+0000 7f3af2ffd640 1 -- 172.21.15.46:0/205256741 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4031930 con 0x7f3af415d330 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 -- 172.21.15.46:0/205256741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af415d330 msgr2=0x7f3af40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 --2- 172.21.15.46:0/205256741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af415d330 0x7f3af40b8b80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097a0 tx=0x7f3ae402b840 comp rx=0 tx=0).stop 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 -- 172.21.15.46:0/205256741 shutdown_connections 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 --2- 172.21.15.46:0/205256741 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af415d330 0x7f3af40b8b80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 -- 172.21.15.46:0/205256741 >> 172.21.15.46:0/205256741 conn(0x7f3af4072020 msgr2=0x7f3af4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 -- 172.21.15.46:0/205256741 shutdown_connections 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.968+0000 7f3afa415640 1 -- 172.21.15.46:0/205256741 wait complete. 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.969+0000 7f3afa415640 1 Processor -- start 2023-12-16T21:14:04.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.969+0000 7f3afa415640 1 -- start start 2023-12-16T21:14:04.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3afa415640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 0x7f3af4152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3afa415640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3af40d3900 0x7f3af4149430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3afa415640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af415d330 0x7f3af4155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3afa415640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40c7830 con 0x7f3af40d2910 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3afa415640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4154480 con 0x7f3af415d330 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3afa415640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40c7a80 con 0x7f3af40d3900 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 0x7f3af4152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 0x7f3af4152c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57604/0 (socket says 172.21.15.46:57604) 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 -- 172.21.15.46:0/2920479547 learned_addr learned my addr 172.21.15.46:0/2920479547 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af89b2640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af415d330 0x7f3af4155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af37fe640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3af40d3900 0x7f3af4149430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 -- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3af40d3900 msgr2=0x7f3af4149430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3af40d3900 0x7f3af4149430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 -- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af415d330 msgr2=0x7f3af4155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af415d330 0x7f3af4155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:04.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.970+0000 7f3af3fff640 1 -- 172.21.15.46:0/2920479547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4009400 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.971+0000 7f3af3fff640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 0x7f3af4152c50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4034040 tx=0x7f3ae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.971+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4032900 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.971+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4032a80 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.971+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af40c7d60 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.971+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4032e30 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.971+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af4152930 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.972+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4069400 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.972+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3ae403b3f0 con 0x7f3af40d2910 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.972+0000 7f3af0ff9640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ad406ae40 0x7f3ad406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:04.974 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.973+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ae40ab730 con 0x7f3af40d2910 2023-12-16T21:14:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.973+0000 7f3af37fe640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ad406ae40 0x7f3ad406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:04.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.974+0000 7f3af37fe640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ad406ae40 0x7f3ad406d300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3af40c6b50 tx=0x7f3adc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:04.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:04.977+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3ae40767e0 con 0x7f3af40d2910 2023-12-16T21:14:05.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.152+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3af40d0030 con 0x7f3af40d2910 2023-12-16T21:14:05.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.152+0000 7f3af0ff9640 1 -- 172.21.15.46:0/2920479547 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ae4079e00 con 0x7f3af40d2910 2023-12-16T21:14:05.152 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:05.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ad406ae40 msgr2=0x7f3ad406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ad406ae40 0x7f3ad406d300 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3af40c6b50 tx=0x7f3adc009040 comp rx=0 tx=0).stop 2023-12-16T21:14:05.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 msgr2=0x7f3af4152c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 0x7f3af4152c50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4034040 tx=0x7f3ae40046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 shutdown_connections 2023-12-16T21:14:05.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ad406ae40 0x7f3ad406d300 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af415d330 0x7f3af4155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3af40d3900 0x7f3af4149430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 --2- 172.21.15.46:0/2920479547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3af40d2910 0x7f3af4152c50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 >> 172.21.15.46:0/2920479547 conn(0x7f3af4072020 msgr2=0x7f3af40b7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 shutdown_connections 2023-12-16T21:14:05.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.154+0000 7f3afa415640 1 -- 172.21.15.46:0/2920479547 wait complete. 2023-12-16T21:14:05.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.275+0000 7feb967e5640 1 Processor -- start 2023-12-16T21:14:05.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.275+0000 7feb967e5640 1 -- start start 2023-12-16T21:14:05.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb967e5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7feb9015c2f0 0x7feb9006eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb967e5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9015ba40 con 0x7feb90140f10 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb967e5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9006f4a0 con 0x7feb9006f040 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb967e5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9015c6d0 con 0x7feb9015c2f0 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb94d82640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7feb90140f10 0x7feb9013f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52324/0 (socket says 172.21.15.46:52324) 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb94d82640 1 -- 172.21.15.46:0/1912177937 learned_addr learned my addr 172.21.15.46:0/1912177937 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb8ffff640 1 --2- 172.21.15.46:0/1912177937 >> v2:172.21.15.121:3300/0 conn(0x7feb9015c2f0 0x7feb9006eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620108664 0 0) 0x7feb9015ba40 con 0x7feb90140f10 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.276+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb78003520 con 0x7feb90140f10 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459896345 0 0) 0x7feb78003520 con 0x7feb90140f10 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 >> v2:172.21.15.121:3300/0 conn(0x7feb9015c2f0 msgr2=0x7feb9006eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 --2- 172.21.15.46:0/1912177937 >> v2:172.21.15.121:3300/0 conn(0x7feb9015c2f0 0x7feb9006eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 >> v1:172.21.15.121:6789/0 conn(0x7feb9006f040 legacy=0x7feb90140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb901416e0 con 0x7feb90140f10 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7feb880019f0 con 0x7feb90140f10 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 >> v1:172.21.15.46:6789/0 conn(0x7feb90140f10 legacy=0x7feb9013f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78003970 0x7feb78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb780047c0 0x7feb78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb78005180 0x7feb7800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7800bff0 con 0x7feb78005180 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb7800c160 con 0x7feb78003970 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb7800c320 con 0x7feb780047c0 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb94d82640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb78005180 0x7feb7800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8ffff640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78003970 0x7feb78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb8f7fe640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb780047c0 0x7feb78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb94d82640 1 -- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb780047c0 msgr2=0x7feb78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb94d82640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb780047c0 0x7feb78004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb94d82640 1 -- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78003970 msgr2=0x7feb78003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb94d82640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78003970 0x7feb78003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.277+0000 7feb94d82640 1 -- 172.21.15.46:0/1912177937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb78003520 con 0x7feb78005180 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.278+0000 7feb94d82640 1 -- 172.21.15.46:0/1912177937 reap_dead start 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.278+0000 7feb94d82640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb78005180 0x7feb7800b980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feb8800a830 tx=0x7feb780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e833cd0220e508e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.278+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb880072d0 con 0x7feb78005180 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.278+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8802f070 con 0x7feb78005180 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.278+0000 7feb8effd640 1 -- 172.21.15.46:0/1912177937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb880076b0 con 0x7feb78005180 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 -- 172.21.15.46:0/1912177937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb78005180 msgr2=0x7feb7800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb78005180 0x7feb7800b980 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7feb8800a830 tx=0x7feb780057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 -- 172.21.15.46:0/1912177937 shutdown_connections 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 --2- 172.21.15.46:0/1912177937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb78005180 0x7feb7800b980 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 -- 172.21.15.46:0/1912177937 >> 172.21.15.46:0/1912177937 conn(0x7feb9006a6b0 msgr2=0x7feb9006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 -- 172.21.15.46:0/1912177937 shutdown_connections 2023-12-16T21:14:05.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.279+0000 7feb967e5640 1 -- 172.21.15.46:0/1912177937 wait complete. 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.280+0000 7feb967e5640 1 Processor -- start 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.280+0000 7feb967e5640 1 -- start start 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb967e5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78005180 0x7feb9013d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb967e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 0x7feb90153f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb967e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb90154490 0x7feb900b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb967e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90152a60 con 0x7feb90158ed0 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb967e5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90152710 con 0x7feb78005180 2023-12-16T21:14:05.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb967e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9013db40 con 0x7feb90154490 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 0x7feb90153f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 0x7feb90153f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57612/0 (socket says 172.21.15.46:57612) 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 -- 172.21.15.46:0/3441652737 learned_addr learned my addr 172.21.15.46:0/3441652737 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 -- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb90154490 msgr2=0x7feb900b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb90154490 0x7feb900b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 -- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78005180 msgr2=0x7feb9013d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78005180 0x7feb9013d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 -- 172.21.15.46:0/3441652737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb90157a80 con 0x7feb90158ed0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.281+0000 7feb8f7fe640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 0x7feb90153f50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb8000e4b0 tx=0x7feb8000e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.282+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800052e0 con 0x7feb90158ed0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.282+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb80005440 con 0x7feb90158ed0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.282+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb901549a0 con 0x7feb90158ed0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.282+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80003830 con 0x7feb90158ed0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.282+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb90154f60 con 0x7feb90158ed0 2023-12-16T21:14:05.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.283+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb90069400 con 0x7feb90158ed0 2023-12-16T21:14:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.283+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7feb80003990 con 0x7feb90158ed0 2023-12-16T21:14:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.283+0000 7feb8cff9640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7feb7c06ae40 0x7feb7c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.283+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb800892a0 con 0x7feb90158ed0 2023-12-16T21:14:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.284+0000 7feb8ffff640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7feb7c06ae40 0x7feb7c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.284+0000 7feb8ffff640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7feb7c06ae40 0x7feb7c06d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb900b12f0 tx=0x7feb84009320 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.288+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7feb800543e0 con 0x7feb90158ed0 2023-12-16T21:14:05.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.465+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feb9013e500 con 0x7feb90158ed0 2023-12-16T21:14:05.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.465+0000 7feb8cff9640 1 -- 172.21.15.46:0/3441652737 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7feb80057a00 con 0x7feb90158ed0 2023-12-16T21:14:05.465 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7feb7c06ae40 msgr2=0x7feb7c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7feb7c06ae40 0x7feb7c06d300 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feb900b12f0 tx=0x7feb84009320 comp rx=0 tx=0).stop 2023-12-16T21:14:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 msgr2=0x7feb90153f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 0x7feb90153f50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feb8000e4b0 tx=0x7feb8000e970 comp rx=0 tx=0).stop 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 shutdown_connections 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7feb7c06ae40 0x7feb7c06d300 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feb90158ed0 0x7feb90153f50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feb90154490 0x7feb900b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 --2- 172.21.15.46:0/3441652737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb78005180 0x7feb9013d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 >> 172.21.15.46:0/3441652737 conn(0x7feb9006a6b0 msgr2=0x7feb9006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 shutdown_connections 2023-12-16T21:14:05.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.467+0000 7feb967e5640 1 -- 172.21.15.46:0/3441652737 wait complete. 2023-12-16T21:14:05.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:05.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.585+0000 7f80b8978640 1 Processor -- start 2023-12-16T21:14:05.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.585+0000 7f80b8978640 1 -- start start 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b8978640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f80b40d2b00 0x7f80b41682a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b8978640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b41614e0 con 0x7f80b4160f70 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b8978640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b41617c0 con 0x7f80b415d2d0 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b8978640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b4161aa0 con 0x7f80b40d2b00 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80abfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f80b4160f70 0x7f80b40cdf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52332/0 (socket says 172.21.15.46:52332) 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80abfff640 1 -- 172.21.15.46:0/4209955302 learned_addr learned my addr 172.21.15.46:0/4209955302 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b2d9d640 1 --2- 172.21.15.46:0/4209955302 >> v2:172.21.15.121:3300/0 conn(0x7f80b40d2b00 0x7f80b41682a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b1d9b640 1 -- 172.21.15.46:0/4209955302 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595265617 0 0) 0x7f80b41614e0 con 0x7f80b4160f70 2023-12-16T21:14:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b1d9b640 1 -- 172.21.15.46:0/4209955302 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a0003520 con 0x7f80b4160f70 2023-12-16T21:14:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b1d9b640 1 -- 172.21.15.46:0/4209955302 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2540435829 0 0) 0x7f80a0003520 con 0x7f80b4160f70 2023-12-16T21:14:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b2d9d640 1 -- 172.21.15.46:0/4209955302 >> v1:172.21.15.121:6789/0 conn(0x7f80b415d2d0 legacy=0x7f80b40b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b2d9d640 1 -- 172.21.15.46:0/4209955302 >> v1:172.21.15.46:6789/0 conn(0x7f80b4160f70 legacy=0x7f80b40cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.586+0000 7f80b2d9d640 1 -- 172.21.15.46:0/4209955302 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4161d80 con 0x7f80b40d2b00 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.587+0000 7f80b2d9d640 1 --2- 172.21.15.46:0/4209955302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b40d2b00 0x7f80b41682a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac02bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ceed5cb94739d7d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b1d9b640 1 -- 172.21.15.46:0/4209955302 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c001670 con 0x7f80b40d2b00 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b1d9b640 1 -- 172.21.15.46:0/4209955302 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac02c7a0 con 0x7f80b40d2b00 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b1d9b640 1 -- 172.21.15.46:0/4209955302 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac02cb30 con 0x7f80b40d2b00 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 -- 172.21.15.46:0/4209955302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b40d2b00 msgr2=0x7f80b41682a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 --2- 172.21.15.46:0/4209955302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b40d2b00 0x7f80b41682a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac02bc70 comp rx=0 tx=0).stop 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 -- 172.21.15.46:0/4209955302 shutdown_connections 2023-12-16T21:14:05.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 --2- 172.21.15.46:0/4209955302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b40d2b00 0x7f80b41682a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 -- 172.21.15.46:0/4209955302 >> 172.21.15.46:0/4209955302 conn(0x7f80b4072230 msgr2=0x7f80b40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 -- 172.21.15.46:0/4209955302 shutdown_connections 2023-12-16T21:14:05.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.588+0000 7f80b8978640 1 -- 172.21.15.46:0/4209955302 wait complete. 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.589+0000 7f80b8978640 1 Processor -- start 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.589+0000 7f80b8978640 1 -- start start 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b8978640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80b40d2b00 0x7f80b4151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b8978640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b415d2d0 0x7f80b40b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b8978640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 0x7f80b41f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b8978640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b415e8e0 con 0x7f80b4160f70 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b8978640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40b3150 con 0x7f80b415d2d0 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b8978640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40b55b0 con 0x7f80b40d2b00 2023-12-16T21:14:05.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 0x7f80b41f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 0x7f80b41f6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57614/0 (socket says 172.21.15.46:57614) 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b2d9d640 1 -- 172.21.15.46:0/2898149103 learned_addr learned my addr 172.21.15.46:0/2898149103 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80abfff640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b415d2d0 0x7f80b40b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.590+0000 7f80b259c640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80b40d2b00 0x7f80b4151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b2d9d640 1 -- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80b40d2b00 msgr2=0x7f80b4151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b2d9d640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80b40d2b00 0x7f80b4151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b2d9d640 1 -- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b415d2d0 msgr2=0x7f80b40b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b2d9d640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b415d2d0 0x7f80b40b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b2d9d640 1 -- 172.21.15.46:0/2898149103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009a70 con 0x7f80b4160f70 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b2d9d640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 0x7f80b41f6340 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f80ac037040 tx=0x7f80ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac034c00 con 0x7f80b4160f70 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac034d80 con 0x7f80b4160f70 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b4154d10 con 0x7f80b4160f70 2023-12-16T21:14:05.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac003bd0 con 0x7f80b4160f70 2023-12-16T21:14:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.591+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b40b5b60 con 0x7f80b4160f70 2023-12-16T21:14:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.592+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b415eae0 con 0x7f80b4160f70 2023-12-16T21:14:05.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.592+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f80ac03b050 con 0x7f80b4160f70 2023-12-16T21:14:05.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.593+0000 7f80aaffd640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f809406ae70 0x7f809406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.593+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80ac0aba60 con 0x7f80b4160f70 2023-12-16T21:14:05.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.593+0000 7f80b259c640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f809406ae70 0x7f809406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.593+0000 7f80b259c640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f809406ae70 0x7f809406d330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000f60 tx=0x7f80a4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.597+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f80b415eae0 con 0x7f80b4160f70 2023-12-16T21:14:05.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.777+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80b415eae0 con 0x7f80b4160f70 2023-12-16T21:14:05.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.778+0000 7f80aaffd640 1 -- 172.21.15.46:0/2898149103 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f80b415eae0 con 0x7f80b4160f70 2023-12-16T21:14:05.778 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:05.778 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:05.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f809406ae70 msgr2=0x7f809406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f809406ae70 0x7f809406d330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000f60 tx=0x7f80a4009040 comp rx=0 tx=0).stop 2023-12-16T21:14:05.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 msgr2=0x7f80b41f6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 0x7f80b41f6340 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f80ac037040 tx=0x7f80ac004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:05.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 shutdown_connections 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f809406ae70 0x7f809406d330 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80b4160f70 0x7f80b41f6340 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80b415d2d0 0x7f80b40b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 --2- 172.21.15.46:0/2898149103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80b40d2b00 0x7f80b4151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 >> 172.21.15.46:0/2898149103 conn(0x7f80b4072230 msgr2=0x7f80b40d3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 shutdown_connections 2023-12-16T21:14:05.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.780+0000 7f80b8978640 1 -- 172.21.15.46:0/2898149103 wait complete. 2023-12-16T21:14:05.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:05.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:05.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.901+0000 7f372f489640 1 Processor -- start 2023-12-16T21:14:05.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.901+0000 7f372f489640 1 -- start start 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372f489640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3728143490 0x7f37280b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372f489640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f372815abf0 0x7f37280b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372f489640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f372815e2a0 con 0x7f372815e730 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372f489640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372815b520 con 0x7f3728143490 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372f489640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3728143870 con 0x7f372815abf0 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3728143490 0x7f37280b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3728143490 0x7f37280b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57630/0 (socket says 172.21.15.46:57630) 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 -- 172.21.15.46:0/4172982609 learned_addr learned my addr 172.21.15.46:0/4172982609 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372da26640 1 --2- 172.21.15.46:0/4172982609 >> v2:172.21.15.121:3300/0 conn(0x7f372815abf0 0x7f37280b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 -- 172.21.15.46:0/4172982609 >> v2:172.21.15.121:3300/0 conn(0x7f372815abf0 msgr2=0x7f37280b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:05.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 --2- 172.21.15.46:0/4172982609 >> v2:172.21.15.121:3300/0 conn(0x7f372815abf0 0x7f37280b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 -- 172.21.15.46:0/4172982609 >> v1:172.21.15.121:6789/0 conn(0x7f372815e730 legacy=0x7f372815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372d225640 1 -- 172.21.15.46:0/4172982609 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372815b240 con 0x7f3728143490 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.902+0000 7f372da26640 1 --2- 172.21.15.46:0/4172982609 >> v2:172.21.15.121:3300/0 conn(0x7f372815abf0 0x7f37280b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.903+0000 7f372d225640 1 --2- 172.21.15.46:0/4172982609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3728143490 0x7f37280b0cf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37280b4ca0 tx=0x7f372402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9b554fff8ff161 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.903+0000 7f371ffff640 1 -- 172.21.15.46:0/4172982609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724031870 con 0x7f3728143490 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.903+0000 7f371ffff640 1 -- 172.21.15.46:0/4172982609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37240319d0 con 0x7f3728143490 2023-12-16T21:14:05.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.903+0000 7f371ffff640 1 -- 172.21.15.46:0/4172982609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724031d60 con 0x7f3728143490 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 -- 172.21.15.46:0/4172982609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3728143490 msgr2=0x7f37280b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 --2- 172.21.15.46:0/4172982609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3728143490 0x7f37280b0cf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37280b4ca0 tx=0x7f372402b910 comp rx=0 tx=0).stop 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 -- 172.21.15.46:0/4172982609 shutdown_connections 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 --2- 172.21.15.46:0/4172982609 >> v2:172.21.15.121:3300/0 conn(0x7f372815abf0 0x7f37280b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 --2- 172.21.15.46:0/4172982609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3728143490 0x7f37280b0cf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 -- 172.21.15.46:0/4172982609 >> 172.21.15.46:0/4172982609 conn(0x7f372806a4d0 msgr2=0x7f372806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 -- 172.21.15.46:0/4172982609 shutdown_connections 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.904+0000 7f372f489640 1 -- 172.21.15.46:0/4172982609 wait complete. 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.905+0000 7f372f489640 1 Processor -- start 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.905+0000 7f372f489640 1 -- start start 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372f489640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3728143490 0x7f37281599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372f489640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f372815abf0 0x7f37280ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372f489640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 0x7f372815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372f489640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3728159f10 con 0x7f372815e730 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372f489640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37280a8570 con 0x7f3728143490 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372f489640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37280a8850 con 0x7f372815abf0 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 0x7f372815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 0x7f372815c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57632/0 (socket says 172.21.15.46:57632) 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 -- 172.21.15.46:0/3217957248 learned_addr learned my addr 172.21.15.46:0/3217957248 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372d225640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3728143490 0x7f37281599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 -- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f372815abf0 msgr2=0x7f37280ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f372815abf0 0x7f37280ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 -- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3728143490 msgr2=0x7f37281599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3728143490 0x7f37281599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 -- 172.21.15.46:0/3217957248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37240094d0 con 0x7f372815e730 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372d225640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3728143490 0x7f37281599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.906+0000 7f372da26640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 0x7f372815c410 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3720007fc0 tx=0x7f372000e4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.907+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720011070 con 0x7f372815e730 2023-12-16T21:14:05.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.907+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372000c0d0 con 0x7f372815e730 2023-12-16T21:14:05.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.907+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37280a8b30 con 0x7f372815e730 2023-12-16T21:14:05.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.907+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372000a530 con 0x7f372815e730 2023-12-16T21:14:05.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.907+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372812b480 con 0x7f372815e730 2023-12-16T21:14:05.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.908+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728069400 con 0x7f372815e730 2023-12-16T21:14:05.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.908+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f372000a690 con 0x7f372815e730 2023-12-16T21:14:05.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.909+0000 7f371dffb640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f370c06ae70 0x7f370c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:05.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.909+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f372008b9a0 con 0x7f372815e730 2023-12-16T21:14:05.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.909+0000 7f372d225640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f370c06ae70 0x7f370c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:05.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.909+0000 7f372d225640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f370c06ae70 0x7f370c06d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3724036000 tx=0x7f372403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:05.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:05.913+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3720056a50 con 0x7f372815e730 2023-12-16T21:14:06.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.087+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37280b2820 con 0x7f372815e730 2023-12-16T21:14:06.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.087+0000 7f371dffb640 1 -- 172.21.15.46:0/3217957248 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f372005a070 con 0x7f372815e730 2023-12-16T21:14:06.087 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f370c06ae70 msgr2=0x7f370c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f370c06ae70 0x7f370c06d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3724036000 tx=0x7f372403c040 comp rx=0 tx=0).stop 2023-12-16T21:14:06.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 msgr2=0x7f372815c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 0x7f372815c410 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3720007fc0 tx=0x7f372000e4e0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 shutdown_connections 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f370c06ae70 0x7f370c06d330 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f372815e730 0x7f372815c410 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f372815abf0 0x7f37280ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 --2- 172.21.15.46:0/3217957248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3728143490 0x7f37281599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 >> 172.21.15.46:0/3217957248 conn(0x7f372806a4d0 msgr2=0x7f372815d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 shutdown_connections 2023-12-16T21:14:06.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.089+0000 7f372f489640 1 -- 172.21.15.46:0/3217957248 wait complete. 2023-12-16T21:14:06.099 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:06.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.209+0000 7f2181664640 1 Processor -- start 2023-12-16T21:14:06.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.209+0000 7f2181664640 1 -- start start 2023-12-16T21:14:06.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f2181664640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f217c15d2d0 0x7f217c0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f2181664640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f2181664640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f217c0cda40 0x7f217c0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f2181664640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217c0d1bb0 con 0x7f217c0cda40 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f2181664640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f217c15d6b0 con 0x7f217c15d2d0 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f2181664640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0ce700 con 0x7f217c0d2040 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f217c0cda40 0x7f217c0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f217c0cda40 0x7f217c0cded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57640/0 (socket says 172.21.15.46:57640) 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 -- 172.21.15.46:0/3360674520 learned_addr learned my addr 172.21.15.46:0/3360674520 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217affd640 1 --2- 172.21.15.46:0/3360674520 >> v2:172.21.15.121:3300/0 conn(0x7f217c15d2d0 0x7f217c0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217a7fc640 1 --2- 172.21.15.46:0/3360674520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 -- 172.21.15.46:0/3360674520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 msgr2=0x7f217c0b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 --2- 172.21.15.46:0/3360674520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 -- 172.21.15.46:0/3360674520 >> v2:172.21.15.121:3300/0 conn(0x7f217c15d2d0 msgr2=0x7f217c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 --2- 172.21.15.46:0/3360674520 >> v2:172.21.15.121:3300/0 conn(0x7f217c15d2d0 0x7f217c0b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217b7fe640 1 -- 172.21.15.46:0/3360674520 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c0d1280 con 0x7f217c0cda40 2023-12-16T21:14:06.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.210+0000 7f217affd640 1 --2- 172.21.15.46:0/3360674520 >> v2:172.21.15.121:3300/0 conn(0x7f217c15d2d0 0x7f217c0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.211+0000 7f217b7fe640 1 --2- 172.21.15.46:0/3360674520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0cded0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f217c06c5d0 tx=0x7f216c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b293c7773dcfe8e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.211+0000 7f2179ffb640 1 -- 172.21.15.46:0/3360674520 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c02ca70 con 0x7f217c0cda40 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.211+0000 7f2179ffb640 1 -- 172.21.15.46:0/3360674520 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f216c02cbd0 con 0x7f217c0cda40 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.211+0000 7f2179ffb640 1 -- 172.21.15.46:0/3360674520 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216c0343f0 con 0x7f217c0cda40 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 -- 172.21.15.46:0/3360674520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 msgr2=0x7f217c0cded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 --2- 172.21.15.46:0/3360674520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0cded0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f217c06c5d0 tx=0x7f216c02bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 -- 172.21.15.46:0/3360674520 shutdown_connections 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 --2- 172.21.15.46:0/3360674520 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0cded0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 --2- 172.21.15.46:0/3360674520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 --2- 172.21.15.46:0/3360674520 >> v2:172.21.15.121:3300/0 conn(0x7f217c15d2d0 0x7f217c0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 -- 172.21.15.46:0/3360674520 >> 172.21.15.46:0/3360674520 conn(0x7f217c072050 msgr2=0x7f217c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:06.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 -- 172.21.15.46:0/3360674520 shutdown_connections 2023-12-16T21:14:06.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.212+0000 7f2181664640 1 -- 172.21.15.46:0/3360674520 wait complete. 2023-12-16T21:14:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.213+0000 7f2181664640 1 Processor -- start 2023-12-16T21:14:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.213+0000 7f2181664640 1 -- start start 2023-12-16T21:14:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f2181664640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f2181664640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f2181664640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f217c15d2d0 0x7f217c152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f2181664640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0c7c00 con 0x7f217c0cda40 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f2181664640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0cb300 con 0x7f217c15d2d0 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f2181664640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217c0cb5a0 con 0x7f217c0d2040 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57644/0 (socket says 172.21.15.46:57644) 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 -- 172.21.15.46:0/403729894 learned_addr learned my addr 172.21.15.46:0/403729894 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217b7fe640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f217c15d2d0 0x7f217c152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217a7fc640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 -- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 msgr2=0x7f217c0b7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 -- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f217c15d2d0 msgr2=0x7f217c152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f217c15d2d0 0x7f217c152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217affd640 1 -- 172.21.15.46:0/403729894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c009b30 con 0x7f217c0cda40 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.214+0000 7f217a7fc640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:06.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.215+0000 7f217affd640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0c6cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f217400d7d0 tx=0x7f217400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.215+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174004030 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.215+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2174004190 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.215+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f217c0b8390 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.215+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174004520 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.215+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c0c7640 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.216+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f217c069400 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.216+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f21740096a0 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.217+0000 7f21737fe640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f215c06ae70 0x7f215c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.217+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2174089760 con 0x7f217c0cda40 2023-12-16T21:14:06.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.217+0000 7f217a7fc640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f215c06ae70 0x7f215c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.217+0000 7f217a7fc640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f215c06ae70 0x7f215c06d330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f217c156590 tx=0x7f2164008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.221+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f21740548a0 con 0x7f217c0cda40 2023-12-16T21:14:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.399+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f217c0cf270 con 0x7f217c0cda40 2023-12-16T21:14:06.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.399+0000 7f21737fe640 1 -- 172.21.15.46:0/403729894 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2174057ec0 con 0x7f217c0cda40 2023-12-16T21:14:06.399 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f215c06ae70 msgr2=0x7f215c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f215c06ae70 0x7f215c06d330 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f217c156590 tx=0x7f2164008040 comp rx=0 tx=0).stop 2023-12-16T21:14:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 msgr2=0x7f217c0c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0c6cf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f217400d7d0 tx=0x7f217400dc90 comp rx=0 tx=0).stop 2023-12-16T21:14:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 shutdown_connections 2023-12-16T21:14:06.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f215c06ae70 0x7f215c06d330 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f217c15d2d0 0x7f217c152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f217c0d2040 0x7f217c0b7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 --2- 172.21.15.46:0/403729894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f217c0cda40 0x7f217c0c6cf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 >> 172.21.15.46:0/403729894 conn(0x7f217c072050 msgr2=0x7f217c0b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 shutdown_connections 2023-12-16T21:14:06.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.401+0000 7f2181664640 1 -- 172.21.15.46:0/403729894 wait complete. 2023-12-16T21:14:06.412 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:06.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.522+0000 7f3e26651640 1 Processor -- start 2023-12-16T21:14:06.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.522+0000 7f3e26651640 1 -- start start 2023-12-16T21:14:06.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e26651640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e2006ea50 0x7f3e2006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e26651640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e200d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e26651640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e20161d20 con 0x7f3e20160fb0 2023-12-16T21:14:06.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e26651640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e2006f370 con 0x7f3e2006ea50 2023-12-16T21:14:06.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e26651640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2015e760 con 0x7f3e200d0ed0 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e2006ea50 0x7f3e2006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e1ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e2006ea50 0x7f3e2006ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57646/0 (socket says 172.21.15.46:57646) 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e1ffff640 1 -- 172.21.15.46:0/749396700 learned_addr learned my addr 172.21.15.46:0/749396700 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e24bee640 1 --2- 172.21.15.46:0/749396700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e200d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.523+0000 7f3e1ffff640 1 -- 172.21.15.46:0/749396700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 msgr2=0x7f3e200d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/749396700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e200d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1ffff640 1 -- 172.21.15.46:0/749396700 >> v1:172.21.15.121:6789/0 conn(0x7f3e20160fb0 legacy=0x7f3e20161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1ffff640 1 -- 172.21.15.46:0/749396700 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e2015d9a0 con 0x7f3e2006ea50 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/749396700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e2006ea50 0x7f3e2006ee30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009870 tx=0x7f3e1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70f27cc4bc241a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1effd640 1 -- 172.21.15.46:0/749396700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14030840 con 0x7f3e2006ea50 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1effd640 1 -- 172.21.15.46:0/749396700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e140309a0 con 0x7f3e2006ea50 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.524+0000 7f3e1effd640 1 -- 172.21.15.46:0/749396700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e14030d30 con 0x7f3e2006ea50 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 -- 172.21.15.46:0/749396700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e2006ea50 msgr2=0x7f3e2006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 --2- 172.21.15.46:0/749396700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e2006ea50 0x7f3e2006ee30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009870 tx=0x7f3e1402b980 comp rx=0 tx=0).stop 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 -- 172.21.15.46:0/749396700 shutdown_connections 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 --2- 172.21.15.46:0/749396700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e200d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 --2- 172.21.15.46:0/749396700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e2006ea50 0x7f3e2006ee30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 -- 172.21.15.46:0/749396700 >> 172.21.15.46:0/749396700 conn(0x7f3e2006a6b0 msgr2=0x7f3e2006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 -- 172.21.15.46:0/749396700 shutdown_connections 2023-12-16T21:14:06.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.525+0000 7f3e26651640 1 -- 172.21.15.46:0/749396700 wait complete. 2023-12-16T21:14:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.526+0000 7f3e26651640 1 Processor -- start 2023-12-16T21:14:06.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.526+0000 7f3e26651640 1 -- start start 2023-12-16T21:14:06.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e26651640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e2006ea50 0x7f3e20132910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e26651640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e201e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e26651640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 0x7f3e200b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e26651640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e20132e50 con 0x7f3e20160fb0 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e26651640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e201e8000 con 0x7f3e2006ea50 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e26651640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e201e8490 con 0x7f3e200d0ed0 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e24bee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 0x7f3e200b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e24bee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 0x7f3e200b1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57662/0 (socket says 172.21.15.46:57662) 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.527+0000 7f3e24bee640 1 -- 172.21.15.46:0/3626195127 learned_addr learned my addr 172.21.15.46:0/3626195127 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e24bee640 1 -- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 msgr2=0x7f3e201e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e24bee640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e201e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e24bee640 1 -- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e2006ea50 msgr2=0x7f3e20132910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e2006ea50 0x7f3e20132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e24bee640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e2006ea50 0x7f3e20132910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e24bee640 1 -- 172.21.15.46:0/3626195127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e140094d0 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e2006ea50 0x7f3e20132910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e24bee640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 0x7f3e200b1fd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e18007d90 tx=0x7f3e1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18014420 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e180145a0 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e201e7150 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18014950 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.528+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e201e76d0 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.529+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e201e8190 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.529+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3e18014ab0 con 0x7f3e20160fb0 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.530+0000 7f3e1cff9640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3e0c06ae40 0x7f3e0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.530+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e18089f70 con 0x7f3e20160fb0 2023-12-16T21:14:06.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.530+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3e0c06ae40 0x7f3e0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.531+0000 7f3e1ffff640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3e0c06ae40 0x7f3e0c06d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3e14036000 tx=0x7f3e1403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.534+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3e180bcea0 con 0x7f3e20160fb0 2023-12-16T21:14:06.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.710+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3e200d1810 con 0x7f3e20160fb0 2023-12-16T21:14:06.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.711+0000 7f3e1cff9640 1 -- 172.21.15.46:0/3626195127 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3e18055020 con 0x7f3e20160fb0 2023-12-16T21:14:06.711 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3e0c06ae40 msgr2=0x7f3e0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3e0c06ae40 0x7f3e0c06d300 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3e14036000 tx=0x7f3e1403b040 comp rx=0 tx=0).stop 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 msgr2=0x7f3e200b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 0x7f3e200b1fd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e18007d90 tx=0x7f3e1800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 shutdown_connections 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3e0c06ae40 0x7f3e0c06d300 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e20160fb0 0x7f3e200b1fd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e200d0ed0 0x7f3e201e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 --2- 172.21.15.46:0/3626195127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e2006ea50 0x7f3e20132910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 >> 172.21.15.46:0/3626195127 conn(0x7f3e2006a6b0 msgr2=0x7f3e2015cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 shutdown_connections 2023-12-16T21:14:06.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.713+0000 7f3e26651640 1 -- 172.21.15.46:0/3626195127 wait complete. 2023-12-16T21:14:06.759 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2023-12-16T21:14:06.759 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-16T21:14:06.760 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.867+0000 7f89fb5a9640 1 Processor -- start 2023-12-16T21:14:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.867+0000 7f89fb5a9640 1 -- start start 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89fb5a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89f4160fb0 0x7f89f4161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89fb5a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f89f40d0ed0 0x7f89f40d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89fb5a9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89f4161d20 con 0x7f89f406ea50 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89fb5a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f415e760 con 0x7f89f40d0ed0 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89fb5a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89f415d9a0 con 0x7f89f4160fb0 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9b46640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f89f40d0ed0 0x7f89f40d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9345640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f89f406ea50 0x7f89f406ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52342/0 (socket says 172.21.15.46:52342) 2023-12-16T21:14:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9345640 1 -- 172.21.15.46:0/1900592212 learned_addr learned my addr 172.21.15.46:0/1900592212 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f8b44640 1 --2- 172.21.15.46:0/1900592212 >> v2:172.21.15.121:3300/0 conn(0x7f89f4160fb0 0x7f89f4161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9b46640 1 -- 172.21.15.46:0/1900592212 >> v2:172.21.15.121:3300/0 conn(0x7f89f4160fb0 msgr2=0x7f89f4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9b46640 1 --2- 172.21.15.46:0/1900592212 >> v2:172.21.15.121:3300/0 conn(0x7f89f4160fb0 0x7f89f4161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9b46640 1 -- 172.21.15.46:0/1900592212 >> v1:172.21.15.46:6789/0 conn(0x7f89f406ea50 legacy=0x7f89f406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89ebfff640 1 -- 172.21.15.46:0/1900592212 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076379271 0 0) 0x7f89f4161d20 con 0x7f89f406ea50 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f9b46640 1 -- 172.21.15.46:0/1900592212 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f415dc80 con 0x7f89f40d0ed0 2023-12-16T21:14:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.868+0000 7f89f8b44640 1 --2- 172.21.15.46:0/1900592212 >> v2:172.21.15.121:3300/0 conn(0x7f89f4160fb0 0x7f89f4161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.869+0000 7f89f9b46640 1 --2- 172.21.15.46:0/1900592212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f40d0ed0 0x7f89f40d1320 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89f40b6690 tx=0x7f89f002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f2bd171ceec747 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.869+0000 7f89ebfff640 1 -- 172.21.15.46:0/1900592212 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f002cbf0 con 0x7f89f40d0ed0 2023-12-16T21:14:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.869+0000 7f89ebfff640 1 -- 172.21.15.46:0/1900592212 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89f002cd50 con 0x7f89f40d0ed0 2023-12-16T21:14:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.869+0000 7f89ebfff640 1 -- 172.21.15.46:0/1900592212 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89f0034660 con 0x7f89f40d0ed0 2023-12-16T21:14:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 -- 172.21.15.46:0/1900592212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f40d0ed0 msgr2=0x7f89f40d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/1900592212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f40d0ed0 0x7f89f40d1320 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f89f40b6690 tx=0x7f89f002be40 comp rx=0 tx=0).stop 2023-12-16T21:14:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 -- 172.21.15.46:0/1900592212 shutdown_connections 2023-12-16T21:14:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/1900592212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f40d0ed0 0x7f89f40d1320 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/1900592212 >> v2:172.21.15.121:3300/0 conn(0x7f89f4160fb0 0x7f89f4161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 -- 172.21.15.46:0/1900592212 >> 172.21.15.46:0/1900592212 conn(0x7f89f406a6b0 msgr2=0x7f89f406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 -- 172.21.15.46:0/1900592212 shutdown_connections 2023-12-16T21:14:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.870+0000 7f89fb5a9640 1 -- 172.21.15.46:0/1900592212 wait complete. 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.871+0000 7f89fb5a9640 1 Processor -- start 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.871+0000 7f89fb5a9640 1 -- start start 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89fb5a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 0x7f89f4154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89fb5a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89f40d0ed0 0x7f89f4153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89fb5a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89f4160fb0 0x7f89f413d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89fb5a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f413daa0 con 0x7f89f406ea50 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89fb5a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f413c1e0 con 0x7f89f40d0ed0 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89fb5a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89f413c4c0 con 0x7f89f4160fb0 2023-12-16T21:14:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 0x7f89f4154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 0x7f89f4154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57676/0 (socket says 172.21.15.46:57676) 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 -- 172.21.15.46:0/2573587904 learned_addr learned my addr 172.21.15.46:0/2573587904 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f8b44640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89f40d0ed0 0x7f89f4153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9b46640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89f4160fb0 0x7f89f413d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 -- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89f4160fb0 msgr2=0x7f89f413d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89f4160fb0 0x7f89f413d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 -- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89f40d0ed0 msgr2=0x7f89f4153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89f40d0ed0 0x7f89f4153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.872+0000 7f89f9345640 1 -- 172.21.15.46:0/2573587904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f0009a70 con 0x7f89f406ea50 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.873+0000 7f89f9345640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 0x7f89f4154d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c430 tx=0x7f89ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.873+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec0074e0 con 0x7f89f406ea50 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.873+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ec007660 con 0x7f89f406ea50 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.873+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f413c7a0 con 0x7f89f406ea50 2023-12-16T21:14:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.873+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ec004a20 con 0x7f89f406ea50 2023-12-16T21:14:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.873+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f413cd60 con 0x7f89f406ea50 2023-12-16T21:14:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.874+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f4069400 con 0x7f89f406ea50 2023-12-16T21:14:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.874+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f89ec004b80 con 0x7f89f406ea50 2023-12-16T21:14:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.875+0000 7f89e9ffb640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f89d806ae40 0x7f89d806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.875+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89ec0894b0 con 0x7f89f406ea50 2023-12-16T21:14:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.875+0000 7f89f8b44640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f89d806ae40 0x7f89d806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:06.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.875+0000 7f89f8b44640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f89d806ae40 0x7f89d806d300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f89f415d0c0 tx=0x7f89e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:06.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:06.879+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f89ec054560 con 0x7f89f406ea50 2023-12-16T21:14:07.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.055+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89f4070d30 con 0x7f89f406ea50 2023-12-16T21:14:07.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.056+0000 7f89e9ffb640 1 -- 172.21.15.46:0/2573587904 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f89ec057b80 con 0x7f89f406ea50 2023-12-16T21:14:07.056 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:07.056 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f89d806ae40 msgr2=0x7f89d806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f89d806ae40 0x7f89d806d300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f89f415d0c0 tx=0x7f89e4008040 comp rx=0 tx=0).stop 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 msgr2=0x7f89f4154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 0x7f89f4154d90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f89ec00c430 tx=0x7f89ec00c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 shutdown_connections 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f89d806ae40 0x7f89d806d300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89f4160fb0 0x7f89f413d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89f40d0ed0 0x7f89f4153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 --2- 172.21.15.46:0/2573587904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89f406ea50 0x7f89f4154d90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 >> 172.21.15.46:0/2573587904 conn(0x7f89f406a6b0 msgr2=0x7f89f40b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 shutdown_connections 2023-12-16T21:14:07.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.058+0000 7f89fb5a9640 1 -- 172.21.15.46:0/2573587904 wait complete. 2023-12-16T21:14:07.068 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:07.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:07.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.178+0000 7f62a3dc9640 1 Processor -- start 2023-12-16T21:14:07.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.179+0000 7f62a3dc9640 1 -- start start 2023-12-16T21:14:07.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.179+0000 7f62a3dc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f629c0b1400 0x7f629c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.179+0000 7f62a3dc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c15e730 0x7f629c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.179+0000 7f62a3dc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f629c15e2a0 con 0x7f629c143490 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.179+0000 7f62a3dc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629c15b520 con 0x7f629c0b1400 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.179+0000 7f62a3dc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c143870 con 0x7f629c15e730 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1364640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f629c0b1400 0x7f629c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1b65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f629c143490 0x7f629c0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52356/0 (socket says 172.21.15.46:52356) 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1b65640 1 -- 172.21.15.46:0/2624405104 learned_addr learned my addr 172.21.15.46:0/2624405104 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a2366640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c15e730 0x7f629c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1364640 1 -- 172.21.15.46:0/2624405104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c15e730 msgr2=0x7f629c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1364640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c15e730 0x7f629c15aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1364640 1 -- 172.21.15.46:0/2624405104 >> v1:172.21.15.46:6789/0 conn(0x7f629c143490 legacy=0x7f629c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a0b63640 1 -- 172.21.15.46:0/2624405104 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614272108 0 0) 0x7f629c15e2a0 con 0x7f629c143490 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a1364640 1 -- 172.21.15.46:0/2624405104 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629c15b240 con 0x7f629c0b1400 2023-12-16T21:14:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.180+0000 7f62a2366640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c15e730 0x7f629c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a1364640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c15a4e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f628c0097a0 tx=0x7f628c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c2a3c858140004 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a0b63640 1 -- 172.21.15.46:0/2624405104 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c031830 con 0x7f629c0b1400 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a0b63640 1 -- 172.21.15.46:0/2624405104 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c031990 con 0x7f629c0b1400 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a0b63640 1 -- 172.21.15.46:0/2624405104 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c031d20 con 0x7f629c0b1400 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 -- 172.21.15.46:0/2624405104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 msgr2=0x7f629c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c15a4e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f628c0097a0 tx=0x7f628c02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 -- 172.21.15.46:0/2624405104 shutdown_connections 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c15e730 0x7f629c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/2624405104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c15a4e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 -- 172.21.15.46:0/2624405104 >> 172.21.15.46:0/2624405104 conn(0x7f629c06a4d0 msgr2=0x7f629c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.181+0000 7f62a3dc9640 1 -- 172.21.15.46:0/2624405104 shutdown_connections 2023-12-16T21:14:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.182+0000 7f62a3dc9640 1 -- 172.21.15.46:0/2624405104 wait complete. 2023-12-16T21:14:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 Processor -- start 2023-12-16T21:14:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 -- start start 2023-12-16T21:14:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c13ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c143490 0x7f629c13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c15e730 0x7f629c15ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c13bb40 con 0x7f629c0b1400 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629c13c510 con 0x7f629c15e730 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a3dc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629c13d0d0 con 0x7f629c143490 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.183+0000 7f62a1b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c13ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c13ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57700/0 (socket says 172.21.15.46:57700) 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 -- 172.21.15.46:0/3483554480 learned_addr learned my addr 172.21.15.46:0/3483554480 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1364640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c143490 0x7f629c13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a2366640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c15e730 0x7f629c15ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 -- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c143490 msgr2=0x7f629c13bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c143490 0x7f629c13bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 -- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c15e730 msgr2=0x7f629c15ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c15e730 0x7f629c15ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 -- 172.21.15.46:0/3483554480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c009400 con 0x7f629c0b1400 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a2366640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c15e730 0x7f629c15ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a1b65640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c13ca60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f629800c4d0 tx=0x7f629800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62980074e0 con 0x7f629c0b1400 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6298007660 con 0x7f629c0b1400 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298004a20 con 0x7f629c0b1400 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629c13d3b0 con 0x7f629c0b1400 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.184+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f629c13b7d0 con 0x7f629c0b1400 2023-12-16T21:14:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.185+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629c15f1f0 con 0x7f629c0b1400 2023-12-16T21:14:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.186+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6298004b80 con 0x7f629c0b1400 2023-12-16T21:14:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.186+0000 7f62927fc640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f628006ae40 0x7f628006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.186+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6298088780 con 0x7f629c0b1400 2023-12-16T21:14:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.186+0000 7f62a1364640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f628006ae40 0x7f628006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.187+0000 7f62a1364640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f628006ae40 0x7f628006d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f628c036000 tx=0x7f628c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.190+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f62980538c0 con 0x7f629c0b1400 2023-12-16T21:14:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.366+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f629c069400 con 0x7f629c0b1400 2023-12-16T21:14:07.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.367+0000 7f62927fc640 1 -- 172.21.15.46:0/3483554480 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f62980180a0 con 0x7f629c0b1400 2023-12-16T21:14:07.366 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f628006ae40 msgr2=0x7f628006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f628006ae40 0x7f628006d300 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f628c036000 tx=0x7f628c03c040 comp rx=0 tx=0).stop 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 msgr2=0x7f629c13ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c13ca60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f629800c4d0 tx=0x7f629800c990 comp rx=0 tx=0).stop 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 shutdown_connections 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f628006ae40 0x7f628006d300 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c15e730 0x7f629c15ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f629c143490 0x7f629c13bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 --2- 172.21.15.46:0/3483554480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f629c0b1400 0x7f629c13ca60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 >> 172.21.15.46:0/3483554480 conn(0x7f629c06a4d0 msgr2=0x7f629c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 shutdown_connections 2023-12-16T21:14:07.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.368+0000 7f62a3dc9640 1 -- 172.21.15.46:0/3483554480 wait complete. 2023-12-16T21:14:07.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.488+0000 7f908c133640 1 Processor -- start 2023-12-16T21:14:07.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f908c133640 1 -- start start 2023-12-16T21:14:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f908c133640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f908406ea50 0x7f908406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f908c133640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f90840d0ed0 0x7f90840d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f908c133640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9084161d20 con 0x7f9084160fb0 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f908c133640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908406f370 con 0x7f908406ea50 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f908c133640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f908415e760 con 0x7f90840d0ed0 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.489+0000 7f9089ecf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f908406ea50 0x7f908406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f9089ecf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f908406ea50 0x7f908406ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57712/0 (socket says 172.21.15.46:57712) 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f9089ecf640 1 -- 172.21.15.46:0/627125945 learned_addr learned my addr 172.21.15.46:0/627125945 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f908a6d0640 1 --2- 172.21.15.46:0/627125945 >> v2:172.21.15.121:3300/0 conn(0x7f90840d0ed0 0x7f90840d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f9089ecf640 1 -- 172.21.15.46:0/627125945 >> v2:172.21.15.121:3300/0 conn(0x7f90840d0ed0 msgr2=0x7f90840d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f9089ecf640 1 --2- 172.21.15.46:0/627125945 >> v2:172.21.15.121:3300/0 conn(0x7f90840d0ed0 0x7f90840d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f9089ecf640 1 -- 172.21.15.46:0/627125945 >> v1:172.21.15.121:6789/0 conn(0x7f9084160fb0 legacy=0x7f9084161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.490+0000 7f9089ecf640 1 -- 172.21.15.46:0/627125945 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908415d9a0 con 0x7f908406ea50 2023-12-16T21:14:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f9089ecf640 1 --2- 172.21.15.46:0/627125945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f908406ea50 0x7f908406ee30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f90841407f0 tx=0x7f907c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fcf0feea7024a5e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f9088ecd640 1 -- 172.21.15.46:0/627125945 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c0317e0 con 0x7f908406ea50 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f9088ecd640 1 -- 172.21.15.46:0/627125945 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907c031940 con 0x7f908406ea50 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f9088ecd640 1 -- 172.21.15.46:0/627125945 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907c031cd0 con 0x7f908406ea50 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 -- 172.21.15.46:0/627125945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f908406ea50 msgr2=0x7f908406ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 --2- 172.21.15.46:0/627125945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f908406ea50 0x7f908406ee30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f90841407f0 tx=0x7f907c02b880 comp rx=0 tx=0).stop 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 -- 172.21.15.46:0/627125945 shutdown_connections 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 --2- 172.21.15.46:0/627125945 >> v2:172.21.15.121:3300/0 conn(0x7f90840d0ed0 0x7f90840d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 --2- 172.21.15.46:0/627125945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f908406ea50 0x7f908406ee30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 -- 172.21.15.46:0/627125945 >> 172.21.15.46:0/627125945 conn(0x7f908406a6b0 msgr2=0x7f908406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 -- 172.21.15.46:0/627125945 shutdown_connections 2023-12-16T21:14:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.491+0000 7f908c133640 1 -- 172.21.15.46:0/627125945 wait complete. 2023-12-16T21:14:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 Processor -- start 2023-12-16T21:14:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 -- start start 2023-12-16T21:14:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f908406ea50 0x7f9084157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 0x7f90841e8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9084160fb0 0x7f908406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90840cab50 con 0x7f90840d0ed0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90841584e0 con 0x7f908406ea50 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f908c133640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90841587c0 con 0x7f9084160fb0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f90896ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 0x7f90841e8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f90896ce640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 0x7f90841e8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57722/0 (socket says 172.21.15.46:57722) 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.493+0000 7f90896ce640 1 -- 172.21.15.46:0/3090985820 learned_addr learned my addr 172.21.15.46:0/3090985820 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f9089ecf640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f908406ea50 0x7f9084157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f908a6d0640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9084160fb0 0x7f908406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f90896ce640 1 -- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9084160fb0 msgr2=0x7f908406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f90896ce640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9084160fb0 0x7f908406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f90896ce640 1 -- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f908406ea50 msgr2=0x7f9084157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f90896ce640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f908406ea50 0x7f9084157ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f90896ce640 1 -- 172.21.15.46:0/3090985820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c0094d0 con 0x7f90840d0ed0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f90896ce640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 0x7f90841e8e00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f907400c430 tx=0x7f907400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074011790 con 0x7f90840d0ed0 2023-12-16T21:14:07.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90740118f0 con 0x7f90840d0ed0 2023-12-16T21:14:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9084158aa0 con 0x7f90840d0ed0 2023-12-16T21:14:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074011ca0 con 0x7f90840d0ed0 2023-12-16T21:14:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.494+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90841e9340 con 0x7f90840d0ed0 2023-12-16T21:14:07.496 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.495+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90840cac90 con 0x7f90840d0ed0 2023-12-16T21:14:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.496+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f907401c460 con 0x7f90840d0ed0 2023-12-16T21:14:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.496+0000 7f907a7fc640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f906806ae40 0x7f906806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.496+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90740895c0 con 0x7f90840d0ed0 2023-12-16T21:14:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.496+0000 7f9089ecf640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f906806ae40 0x7f906806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.497+0000 7f9089ecf640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f906806ae40 0x7f906806d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f907c036000 tx=0x7f907c03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.500+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f90840cac90 con 0x7f90840d0ed0 2023-12-16T21:14:07.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.676+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90840cac90 con 0x7f90840d0ed0 2023-12-16T21:14:07.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.676+0000 7f907a7fc640 1 -- 172.21.15.46:0/3090985820 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90840cac90 con 0x7f90840d0ed0 2023-12-16T21:14:07.676 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f906806ae40 msgr2=0x7f906806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f906806ae40 0x7f906806d300 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f907c036000 tx=0x7f907c03d040 comp rx=0 tx=0).stop 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 msgr2=0x7f90841e8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 0x7f90841e8e00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f907400c430 tx=0x7f907400c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 shutdown_connections 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f906806ae40 0x7f906806d300 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90840d0ed0 0x7f90841e8e00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9084160fb0 0x7f908406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 --2- 172.21.15.46:0/3090985820 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f908406ea50 0x7f9084157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 >> 172.21.15.46:0/3090985820 conn(0x7f908406a6b0 msgr2=0x7f908406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 shutdown_connections 2023-12-16T21:14:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.678+0000 7f908c133640 1 -- 172.21.15.46:0/3090985820 wait complete. 2023-12-16T21:14:07.688 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.798+0000 7fcefb94c640 1 Processor -- start 2023-12-16T21:14:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.798+0000 7fcefb94c640 1 -- start start 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcefb94c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcef40b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcefb94c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcef40ce650 0x7fcef40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcefb94c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef41614e0 con 0x7fcef4160f70 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcefb94c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef41617c0 con 0x7fcef40ce650 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcefb94c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4161aa0 con 0x7fcef415d2d0 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcef8ee7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcef4160f70 0x7fcef40cdf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52370/0 (socket says 172.21.15.46:52370) 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcef8ee7640 1 -- 172.21.15.46:0/3244392320 learned_addr learned my addr 172.21.15.46:0/3244392320 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcef96e8640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcef40b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fcef9ee9640 1 --2- 172.21.15.46:0/3244392320 >> v2:172.21.15.121:3300/0 conn(0x7fcef40ce650 0x7fcef40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585341062 0 0) 0x7fcef41614e0 con 0x7fcef4160f70 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.799+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee0003520 con 0x7fcef4160f70 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3530232392 0 0) 0x7fcee0003520 con 0x7fcef4160f70 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 msgr2=0x7fcef40b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcef40b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 >> v2:172.21.15.121:3300/0 conn(0x7fcef40ce650 msgr2=0x7fcef40d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 --2- 172.21.15.46:0/3244392320 >> v2:172.21.15.121:3300/0 conn(0x7fcef40ce650 0x7fcef40d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef4161d80 con 0x7fcef4160f70 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fcee4001670 con 0x7fcef4160f70 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef96e8640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcef40b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 >> v1:172.21.15.46:6789/0 conn(0x7fcef4160f70 legacy=0x7fcef40cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcee00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee0004d00 0x7fcee0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef41617c0 con 0x7fcee0004d00 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4161aa0 con 0x7fcee0003970 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee000b980 con 0x7fcef415d2d0 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef9ee9640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee0004d00 0x7fcee0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef96e8640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef8ee7640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcee00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef9ee9640 1 -- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 msgr2=0x7fcee00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef9ee9640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcef415d2d0 0x7fcee00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef9ee9640 1 -- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee0003970 msgr2=0x7fcee0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef9ee9640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee0003970 0x7fcee0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.800+0000 7fcef9ee9640 1 -- 172.21.15.46:0/3244392320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec009400 con 0x7fcee0004d00 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.801+0000 7fcef9ee9640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee0004d00 0x7fcee0005180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009fc0 tx=0x7fcee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af94170b889c4d1d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.801+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef002cc60 con 0x7fcee0004d00 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.801+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef000f040 con 0x7fcee0004d00 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.801+0000 7fceebfff640 1 -- 172.21.15.46:0/3244392320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0034440 con 0x7fcee0004d00 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcefb94c640 1 -- 172.21.15.46:0/3244392320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee0004d00 msgr2=0x7fcee0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcefb94c640 1 --2- 172.21.15.46:0/3244392320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcee0004d00 0x7fcee0005180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009fc0 tx=0x7fcee00057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcef9ee9640 1 -- 172.21.15.46:0/3244392320 reap_dead start 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcefb94c640 1 -- 172.21.15.46:0/3244392320 shutdown_connections 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcefb94c640 1 -- 172.21.15.46:0/3244392320 >> 172.21.15.46:0/3244392320 conn(0x7fcef4072230 msgr2=0x7fcef40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcefb94c640 1 -- 172.21.15.46:0/3244392320 shutdown_connections 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.802+0000 7fcefb94c640 1 -- 172.21.15.46:0/3244392320 wait complete. 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.803+0000 7fcefb94c640 1 Processor -- start 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.803+0000 7fcefb94c640 1 -- start start 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcee0004d00 0x7fcef4158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 0x7fcef4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcef40b3540 0x7fcef40b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4158c80 con 0x7fcef4160570 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef40b3ee0 con 0x7fcef40b3540 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef415da50 con 0x7fcee0004d00 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 0x7fcef4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 0x7fcef4159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57742/0 (socket says 172.21.15.46:57742) 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 -- 172.21.15.46:0/1604297880 learned_addr learned my addr 172.21.15.46:0/1604297880 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 -- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcee0004d00 msgr2=0x7fcef4158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcee0004d00 0x7fcef4158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 -- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcef40b3540 msgr2=0x7fcef40b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcef40b3540 0x7fcef40b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 -- 172.21.15.46:0/1604297880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef415dd30 con 0x7fcef4160570 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcef8ee7640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 0x7fcef4159ce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcee4007340 tx=0x7fcee4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400f900 con 0x7fcef4160570 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee400faa0 con 0x7fcef4160570 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef415e010 con 0x7fcef4160570 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.804+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee400fe50 con 0x7fcef4160570 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.805+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef415a220 con 0x7fcef4160570 2023-12-16T21:14:07.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.806+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef415e530 con 0x7fcef4160570 2023-12-16T21:14:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.806+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fcee401d460 con 0x7fcef4160570 2023-12-16T21:14:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.806+0000 7fcee9ffb640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fced806ae40 0x7fced806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.807+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcee40898a0 con 0x7fcef4160570 2023-12-16T21:14:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.807+0000 7fcef96e8640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fced806ae40 0x7fced806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:07.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.807+0000 7fcef96e8640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fced806ae40 0x7fced806d300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fceec009bd0 tx=0x7fceec006a60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:07.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.811+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcef415e530 con 0x7fcef4160570 2023-12-16T21:14:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.991+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcef415e530 con 0x7fcef4160570 2023-12-16T21:14:07.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.991+0000 7fcee9ffb640 1 -- 172.21.15.46:0/1604297880 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcef415e530 con 0x7fcef4160570 2023-12-16T21:14:07.991 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fced806ae40 msgr2=0x7fced806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fced806ae40 0x7fced806d300 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fceec009bd0 tx=0x7fceec006a60 comp rx=0 tx=0).stop 2023-12-16T21:14:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 msgr2=0x7fcef4159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:07.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 0x7fcef4159ce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcee4007340 tx=0x7fcee4007800 comp rx=0 tx=0).stop 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 shutdown_connections 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fced806ae40 0x7fced806d300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcef4160570 0x7fcef4159ce0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcef40b3540 0x7fcef40b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 --2- 172.21.15.46:0/1604297880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcee0004d00 0x7fcef4158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 >> 172.21.15.46:0/1604297880 conn(0x7fcef4072230 msgr2=0x7fcef40c6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.993+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 shutdown_connections 2023-12-16T21:14:07.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:07.994+0000 7fcefb94c640 1 -- 172.21.15.46:0/1604297880 wait complete. 2023-12-16T21:14:08.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2023-12-16T21:14:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.111+0000 7fa4909c5640 1 Processor -- start 2023-12-16T21:14:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.111+0000 7fa4909c5640 1 -- start start 2023-12-16T21:14:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa4909c5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa4909c5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c0ae910 con 0x7fa48c15d2b0 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa4909c5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c0b2200 con 0x7fa48c0ae480 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa4909c5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c06df90 con 0x7fa48c072390 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa48a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa48c15d2b0 0x7fa48c071c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52380/0 (socket says 172.21.15.46:52380) 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa48a59c640 1 -- 172.21.15.46:0/102722300 learned_addr learned my addr 172.21.15.46:0/102722300 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.112+0000 7fa48ad9d640 1 --2- 172.21.15.46:0/102722300 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254770474 0 0) 0x7fa48c0ae910 con 0x7fa48c15d2b0 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa478003520 con 0x7fa48c15d2b0 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723332829 0 0) 0x7fa48c0b2200 con 0x7fa48c0ae480 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa48c0ae910 con 0x7fa48c0ae480 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2356658702 0 0) 0x7fa478003520 con 0x7fa48c15d2b0 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48ad9d640 1 -- 172.21.15.46:0/102722300 >> v1:172.21.15.121:6789/0 conn(0x7fa48c0ae480 legacy=0x7fa48c06d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48ad9d640 1 -- 172.21.15.46:0/102722300 >> v1:172.21.15.46:6789/0 conn(0x7fa48c15d2b0 legacy=0x7fa48c071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.113+0000 7fa48ad9d640 1 -- 172.21.15.46:0/102722300 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c0b24e0 con 0x7fa48c072390 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa48ad9d640 1 --2- 172.21.15.46:0/102722300 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c06da50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009dc0 tx=0x7fa47c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76c2a4106e39a055 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa484001670 con 0x7fa48c072390 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c02c880 con 0x7fa48c072390 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa48959a640 1 -- 172.21.15.46:0/102722300 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c02cbf0 con 0x7fa48c072390 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa4909c5640 1 -- 172.21.15.46:0/102722300 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 msgr2=0x7fa48c06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa4909c5640 1 --2- 172.21.15.46:0/102722300 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c06da50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa47c009dc0 tx=0x7fa47c02bd50 comp rx=0 tx=0).stop 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa4909c5640 1 -- 172.21.15.46:0/102722300 shutdown_connections 2023-12-16T21:14:08.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa4909c5640 1 --2- 172.21.15.46:0/102722300 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c06da50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.114+0000 7fa4909c5640 1 -- 172.21.15.46:0/102722300 >> 172.21.15.46:0/102722300 conn(0x7fa48c0b4720 msgr2=0x7fa48c0b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.115+0000 7fa4909c5640 1 -- 172.21.15.46:0/102722300 shutdown_connections 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.115+0000 7fa4909c5640 1 -- 172.21.15.46:0/102722300 wait complete. 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.116+0000 7fa4909c5640 1 Processor -- start 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.116+0000 7fa4909c5640 1 -- start start 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 0x7fa48c1f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa48c149c00 0x7fa48c09a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1498a0 con 0x7fa48c15d2b0 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c148800 con 0x7fa48c149c00 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c148ae0 con 0x7fa48c072390 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 0x7fa48c1f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 0x7fa48c1f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57754/0 (socket says 172.21.15.46:57754) 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 -- 172.21.15.46:0/3887134929 learned_addr learned my addr 172.21.15.46:0/3887134929 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa48a59c640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa48ad9d640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa48c149c00 0x7fa48c09a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 -- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 msgr2=0x7fa48c147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 -- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa48c149c00 msgr2=0x7fa48c09a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa48c149c00 0x7fa48c09a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 -- 172.21.15.46:0/3887134929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c009a70 con 0x7fa48c15d2b0 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa48ad9d640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa48c149c00 0x7fa48c09a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa489d9b640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 0x7fa48c1f2330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa47400c490 tx=0x7fa47400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4740052d0 con 0x7fa48c15d2b0 2023-12-16T21:14:08.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa474005430 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c148dc0 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.117+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa474012870 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.118+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c149380 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.118+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c0b2950 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.119+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa4740129d0 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.119+0000 7fa482ffd640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa46c06ae40 0x7fa46c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.119+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa474089210 con 0x7fa48c15d2b0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.120+0000 7fa48a59c640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa46c06ae40 0x7fa46c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.120+0000 7fa48a59c640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa46c06ae40 0x7fa46c06d300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa4840011e0 tx=0x7fa484001690 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:08.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.123+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa48c0b2950 con 0x7fa48c15d2b0 2023-12-16T21:14:08.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.370+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) v1 -- 0x7fa48c0b2950 con 0x7fa48c15d2b0 2023-12-16T21:14:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.630+0000 7fa482ffd640 1 -- 172.21.15.46:0/3887134929 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v55) v1 ==== 150+0+0 (secure 0 0 0) 0x7fa48c0b2950 con 0x7fa48c15d2b0 2023-12-16T21:14:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa46c06ae40 msgr2=0x7fa46c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa46c06ae40 0x7fa46c06d300 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa4840011e0 tx=0x7fa484001690 comp rx=0 tx=0).stop 2023-12-16T21:14:08.633 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 msgr2=0x7fa48c1f2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 0x7fa48c1f2330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa47400c490 tx=0x7fa47400c950 comp rx=0 tx=0).stop 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 shutdown_connections 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa46c06ae40 0x7fa46c06d300 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa48c15d2b0 0x7fa48c1f2330 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa48c149c00 0x7fa48c09a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 --2- 172.21.15.46:0/3887134929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa48c072390 0x7fa48c147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 >> 172.21.15.46:0/3887134929 conn(0x7fa48c0b4720 msgr2=0x7fa48c06c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 shutdown_connections 2023-12-16T21:14:08.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.633+0000 7fa4909c5640 1 -- 172.21.15.46:0/3887134929 wait complete. 2023-12-16T21:14:08.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-16T21:14:08.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.753+0000 7f40887c9640 1 Processor -- start 2023-12-16T21:14:08.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.753+0000 7f40887c9640 1 -- start start 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f40887c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f408006ea50 0x7f408006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f40887c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f4080161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f40887c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f40800b3ff0 0x7f40800b4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f40887c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4080161d20 con 0x7f408006ea50 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f40887c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f408006f370 con 0x7f40800b3ff0 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f40887c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408015e760 con 0x7f4080160fb0 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f408006ea50 0x7f408006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f408006ea50 0x7f408006ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57770/0 (socket says 172.21.15.46:57770) 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 -- 172.21.15.46:0/2519066230 learned_addr learned my addr 172.21.15.46:0/2519066230 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4085d64640 1 --2- 172.21.15.46:0/2519066230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f4080161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086d66640 1 --2- 172.21.15.46:0/2519066230 >> v2:172.21.15.121:3300/0 conn(0x7f40800b3ff0 0x7f40800b4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 -- 172.21.15.46:0/2519066230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 msgr2=0x7f4080161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 --2- 172.21.15.46:0/2519066230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f4080161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 -- 172.21.15.46:0/2519066230 >> v2:172.21.15.121:3300/0 conn(0x7f40800b3ff0 msgr2=0x7f40800b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 --2- 172.21.15.46:0/2519066230 >> v2:172.21.15.121:3300/0 conn(0x7f40800b3ff0 0x7f40800b4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.754+0000 7f4086565640 1 -- 172.21.15.46:0/2519066230 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408015d9a0 con 0x7f408006ea50 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.755+0000 7f4086565640 1 --2- 172.21.15.46:0/2519066230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408006ee30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f40800b6690 tx=0x7f407802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be58e37e822120b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.755+0000 7f4085563640 1 -- 172.21.15.46:0/2519066230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4078031870 con 0x7f408006ea50 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.755+0000 7f4085563640 1 -- 172.21.15.46:0/2519066230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40780319d0 con 0x7f408006ea50 2023-12-16T21:14:08.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.755+0000 7f4085563640 1 -- 172.21.15.46:0/2519066230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4078031d60 con 0x7f408006ea50 2023-12-16T21:14:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 -- 172.21.15.46:0/2519066230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 msgr2=0x7f408006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 --2- 172.21.15.46:0/2519066230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408006ee30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f40800b6690 tx=0x7f407802b910 comp rx=0 tx=0).stop 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 -- 172.21.15.46:0/2519066230 shutdown_connections 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 --2- 172.21.15.46:0/2519066230 >> v2:172.21.15.121:3300/0 conn(0x7f40800b3ff0 0x7f40800b4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 --2- 172.21.15.46:0/2519066230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f4080161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 --2- 172.21.15.46:0/2519066230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408006ee30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 -- 172.21.15.46:0/2519066230 >> 172.21.15.46:0/2519066230 conn(0x7f408006a6b0 msgr2=0x7f408006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 -- 172.21.15.46:0/2519066230 shutdown_connections 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.756+0000 7f40887c9640 1 -- 172.21.15.46:0/2519066230 wait complete. 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.757+0000 7f40887c9640 1 Processor -- start 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.757+0000 7f40887c9640 1 -- start start 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f40887c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408015a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f40887c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40800b3ff0 0x7f40801e8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f40887c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f40801e9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f40887c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4080158f60 con 0x7f408006ea50 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f40887c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4080158390 con 0x7f40800b3ff0 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f40887c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4080158630 con 0x7f4080160fb0 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408015a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408015a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57776/0 (socket says 172.21.15.46:57776) 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 -- 172.21.15.46:0/3851993947 learned_addr learned my addr 172.21.15.46:0/3851993947 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 -- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 msgr2=0x7f40801e9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f40801e9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 -- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40800b3ff0 msgr2=0x7f40801e8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40800b3ff0 0x7f40801e8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 -- 172.21.15.46:0/3851993947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40780094d0 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.758+0000 7f4086565640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408015a140 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f407802b950 tx=0x7f40780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.759+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407803ad30 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.759+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407803ae90 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.759+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4080158910 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.759+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4078044630 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.759+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408015a680 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.760+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40800b53c0 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.760+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f407804d450 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.760+0000 7f407effd640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f406c06ae40 0x7f406c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.761+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f407802d040 con 0x7f408006ea50 2023-12-16T21:14:08.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.761+0000 7f4085d64640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f406c06ae40 0x7f406c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.761+0000 7f4085d64640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f406c06ae40 0x7f406c06d300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f40800b3fc0 tx=0x7f4074008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:08.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.765+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4078076c70 con 0x7f408006ea50 2023-12-16T21:14:08.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.942+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4080069400 con 0x7f408006ea50 2023-12-16T21:14:08.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.945+0000 7f407effd640 1 -- 172.21.15.46:0/3851993947 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4078035020 con 0x7f408006ea50 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f406c06ae40 msgr2=0x7f406c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f406c06ae40 0x7f406c06d300 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f40800b3fc0 tx=0x7f4074008040 comp rx=0 tx=0).stop 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 msgr2=0x7f408015a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408015a140 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f407802b950 tx=0x7f40780046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 shutdown_connections 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f406c06ae40 0x7f406c06d300 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4080160fb0 0x7f40801e9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40800b3ff0 0x7f40801e8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 --2- 172.21.15.46:0/3851993947 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f408006ea50 0x7f408015a140 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 >> 172.21.15.46:0/3851993947 conn(0x7f408006a6b0 msgr2=0x7f408015c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 shutdown_connections 2023-12-16T21:14:08.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:08.947+0000 7f40887c9640 1 -- 172.21.15.46:0/3851993947 wait complete. 2023-12-16T21:14:08.957 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-16T21:14:08.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:14:09.009 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:14:09.056 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-16T21:14:09.056 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:09.057 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:09.073 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:09.073 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:09.129 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.275+0000 7f024b044640 1 Processor -- start 2023-12-16T21:14:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.275+0000 7f024b044640 1 -- start start 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f024b044640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02440b1400 0x7f024415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f024b044640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024415e2a0 con 0x7f0244143490 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f024b044640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024415b240 con 0x7f024415e730 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f024b044640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024415b520 con 0x7f02440b1400 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f0243fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f02440b1400 0x7f024415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f0248de0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0244143490 0x7f02440b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52394/0 (socket says 172.21.15.46:52394) 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f0248de0640 1 -- 172.21.15.46:0/671501097 learned_addr learned my addr 172.21.15.46:0/671501097 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f0243fff640 1 -- 172.21.15.46:0/671501097 >> v1:172.21.15.121:6789/0 conn(0x7f024415e730 legacy=0x7f024415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f0243fff640 1 -- 172.21.15.46:0/671501097 >> v1:172.21.15.46:6789/0 conn(0x7f0244143490 legacy=0x7f02440b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f02437fe640 1 -- 172.21.15.46:0/671501097 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698489169 0 0) 0x7f024415e2a0 con 0x7f0244143490 2023-12-16T21:14:09.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.276+0000 7f0243fff640 1 -- 172.21.15.46:0/671501097 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f024415dc50 con 0x7f02440b1400 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.277+0000 7f0243fff640 1 --2- 172.21.15.46:0/671501097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415a4e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02380097a0 tx=0x7f023802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847d39026060d405 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.277+0000 7f02437fe640 1 -- 172.21.15.46:0/671501097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238031830 con 0x7f02440b1400 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.277+0000 7f02437fe640 1 -- 172.21.15.46:0/671501097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0238031990 con 0x7f02440b1400 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.277+0000 7f02437fe640 1 -- 172.21.15.46:0/671501097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238031d20 con 0x7f02440b1400 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 -- 172.21.15.46:0/671501097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 msgr2=0x7f024415a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 --2- 172.21.15.46:0/671501097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415a4e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f02380097a0 tx=0x7f023802b8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 -- 172.21.15.46:0/671501097 shutdown_connections 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 --2- 172.21.15.46:0/671501097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415a4e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 -- 172.21.15.46:0/671501097 >> 172.21.15.46:0/671501097 conn(0x7f024406a4d0 msgr2=0x7f024406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 -- 172.21.15.46:0/671501097 shutdown_connections 2023-12-16T21:14:09.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.278+0000 7f024b044640 1 -- 172.21.15.46:0/671501097 wait complete. 2023-12-16T21:14:09.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.279+0000 7f024b044640 1 Processor -- start 2023-12-16T21:14:09.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.279+0000 7f024b044640 1 -- start start 2023-12-16T21:14:09.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f024b044640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f024b044640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0244143490 0x7f0244164290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f024b044640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f024415e730 0x7f02441f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f024b044640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02441f31a0 con 0x7f02440b1400 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f024b044640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024415d880 con 0x7f0244143490 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f024b044640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024413d780 con 0x7f024415e730 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57802/0 (socket says 172.21.15.46:57802) 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 -- 172.21.15.46:0/4099953773 learned_addr learned my addr 172.21.15.46:0/4099953773 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 -- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f024415e730 msgr2=0x7f02441f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f024415e730 0x7f02441f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 -- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0244143490 msgr2=0x7f0244164290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0244143490 0x7f0244164290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 -- 172.21.15.46:0/4099953773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0238009400 con 0x7f02440b1400 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.280+0000 7f0248de0640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415d340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c890 tx=0x7f023c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.281+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c00a640 con 0x7f02440b1400 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.281+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f023c00a7c0 con 0x7f02440b1400 2023-12-16T21:14:09.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.281+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023c00ab70 con 0x7f02440b1400 2023-12-16T21:14:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.281+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024413d9d0 con 0x7f02440b1400 2023-12-16T21:14:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.281+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02441647d0 con 0x7f02440b1400 2023-12-16T21:14:09.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.282+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02441649c0 con 0x7f02440b1400 2023-12-16T21:14:09.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.282+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f023c00acd0 con 0x7f02440b1400 2023-12-16T21:14:09.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.282+0000 7f02417fa640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f023006ae40 0x7f023006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.283+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f023c088ab0 con 0x7f02440b1400 2023-12-16T21:14:09.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.283+0000 7f0243fff640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f023006ae40 0x7f023006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.283+0000 7f0243fff640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f023006ae40 0x7f023006d300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0238036000 tx=0x7f0238030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.287+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f023c053bf0 con 0x7f02440b1400 2023-12-16T21:14:09.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.463+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02441f32e0 con 0x7f02440b1400 2023-12-16T21:14:09.464 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.464+0000 7f02417fa640 1 -- 172.21.15.46:0/4099953773 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f023c018020 con 0x7f02440b1400 2023-12-16T21:14:09.464 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:09.464 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:09.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f023006ae40 msgr2=0x7f023006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f023006ae40 0x7f023006d300 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0238036000 tx=0x7f0238030000 comp rx=0 tx=0).stop 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 msgr2=0x7f024415d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415d340 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c890 tx=0x7f023c00cd50 comp rx=0 tx=0).stop 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 shutdown_connections 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f023006ae40 0x7f023006d300 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f024415e730 0x7f02441f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0244143490 0x7f0244164290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 --2- 172.21.15.46:0/4099953773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02440b1400 0x7f024415d340 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 >> 172.21.15.46:0/4099953773 conn(0x7f024406a4d0 msgr2=0x7f02440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 shutdown_connections 2023-12-16T21:14:09.469 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.467+0000 7f024b044640 1 -- 172.21.15.46:0/4099953773 wait complete. 2023-12-16T21:14:09.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:09.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:09.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.588+0000 7f094fd7c640 1 Processor -- start 2023-12-16T21:14:09.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.588+0000 7f094fd7c640 1 -- start start 2023-12-16T21:14:09.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094fd7c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094fd7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0948160d20 0x7f09480cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094fd7c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09480d03f0 con 0x7f09480cffe0 2023-12-16T21:14:09.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094fd7c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09480b4430 con 0x7f0948160d20 2023-12-16T21:14:09.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094fd7c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480d1ad0 con 0x7f0948160940 2023-12-16T21:14:09.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0948160d20 0x7f09480cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0948160d20 0x7f09480cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57810/0 (socket says 172.21.15.46:57810) 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 -- 172.21.15.46:0/327099198 learned_addr learned my addr 172.21.15.46:0/327099198 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094db18640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 -- 172.21.15.46:0/327099198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 msgr2=0x7f09480b1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480b1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 -- 172.21.15.46:0/327099198 >> v1:172.21.15.121:6789/0 conn(0x7f09480cffe0 legacy=0x7f09480b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094d317640 1 -- 172.21.15.46:0/327099198 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09480b46d0 con 0x7f0948160d20 2023-12-16T21:14:09.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.589+0000 7f094db18640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:09.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.590+0000 7f094d317640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480cfaa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f093c009870 tx=0x7f093c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5352ca7973c6c93 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.590+0000 7f094cb16640 1 -- 172.21.15.46:0/327099198 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c031640 con 0x7f0948160d20 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.590+0000 7f094cb16640 1 -- 172.21.15.46:0/327099198 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093c0317a0 con 0x7f0948160d20 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.590+0000 7f094cb16640 1 -- 172.21.15.46:0/327099198 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093c031b30 con 0x7f0948160d20 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 -- 172.21.15.46:0/327099198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 msgr2=0x7f09480cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480cfaa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f093c009870 tx=0x7f093c02b8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 -- 172.21.15.46:0/327099198 shutdown_connections 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480cfaa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 --2- 172.21.15.46:0/327099198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480b1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 -- 172.21.15.46:0/327099198 >> 172.21.15.46:0/327099198 conn(0x7f094809c470 msgr2=0x7f09480d3140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 -- 172.21.15.46:0/327099198 shutdown_connections 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.591+0000 7f094fd7c640 1 -- 172.21.15.46:0/327099198 wait complete. 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.592+0000 7f094fd7c640 1 Processor -- start 2023-12-16T21:14:09.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.592+0000 7f094fd7c640 1 -- start start 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094fd7c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480cffe0 0x7f09480b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094fd7c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094fd7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094fd7c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09480c9bb0 con 0x7f0948160d20 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094fd7c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094815ce60 con 0x7f09480cffe0 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094fd7c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09480b40c0 con 0x7f0948160940 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57818/0 (socket says 172.21.15.46:57818) 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 -- 172.21.15.46:0/202033272 learned_addr learned my addr 172.21.15.46:0/202033272 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094db18640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480cffe0 0x7f09480b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094d317640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 -- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 msgr2=0x7f09480c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 -- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480cffe0 msgr2=0x7f09480b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480cffe0 0x7f09480b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094e319640 1 -- 172.21.15.46:0/202033272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c0094d0 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.593+0000 7f094db18640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480cffe0 0x7f09480b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.594+0000 7f094e319640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480c8cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f093800c410 tx=0x7f093800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.594+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938011790 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.594+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09380118f0 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.594+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094815d0b0 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.594+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938011ce0 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.594+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09480c97d0 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.595+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09480d2dc0 con 0x7f0948160d20 2023-12-16T21:14:09.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.595+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f093801c460 con 0x7f0948160d20 2023-12-16T21:14:09.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.596+0000 7f09467fc640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f093406ae40 0x7f093406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.596+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0938089300 con 0x7f0948160d20 2023-12-16T21:14:09.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.596+0000 7f094db18640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f093406ae40 0x7f093406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.597+0000 7f094db18640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f093406ae40 0x7f093406d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f094000e550 tx=0x7f0940007ab0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.600+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f09380543a0 con 0x7f0948160d20 2023-12-16T21:14:09.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.775+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0948069320 con 0x7f0948160d20 2023-12-16T21:14:09.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.776+0000 7f09467fc640 1 -- 172.21.15.46:0/202033272 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f09380579c0 con 0x7f0948160d20 2023-12-16T21:14:09.775 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:09.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f093406ae40 msgr2=0x7f093406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f093406ae40 0x7f093406d300 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f094000e550 tx=0x7f0940007ab0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 msgr2=0x7f09480c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480c8cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f093800c410 tx=0x7f093800c8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 shutdown_connections 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f093406ae40 0x7f093406d300 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0948160d20 0x7f09480c8cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0948160940 0x7f09480c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 --2- 172.21.15.46:0/202033272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480cffe0 0x7f09480b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.777+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 >> 172.21.15.46:0/202033272 conn(0x7f094809c470 msgr2=0x7f09480d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.778+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 shutdown_connections 2023-12-16T21:14:09.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.778+0000 7f094fd7c640 1 -- 172.21.15.46:0/202033272 wait complete. 2023-12-16T21:14:09.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.898+0000 7ff0a1790640 1 Processor -- start 2023-12-16T21:14:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.898+0000 7ff0a1790640 1 -- start start 2023-12-16T21:14:09.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff0a1790640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff09c146840 0x7ff09c0b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff0a1790640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c146fd0 con 0x7ff09c06f1c0 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff0a1790640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c06c180 con 0x7ff09c146460 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff0a1790640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09c06c460 con 0x7ff09c146840 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff09a7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff09c06f1c0 0x7ff09c15db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52410/0 (socket says 172.21.15.46:52410) 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff09a7fc640 1 -- 172.21.15.46:0/3702055604 learned_addr learned my addr 172.21.15.46:0/3702055604 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396182273 0 0) 0x7ff09c146fd0 con 0x7ff09c06f1c0 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff088003520 con 0x7ff09c06f1c0 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.899+0000 7ff09b7fe640 1 --2- 172.21.15.46:0/3702055604 >> v2:172.21.15.121:3300/0 conn(0x7ff09c146840 0x7ff09c0b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1937144469 0 0) 0x7ff088003520 con 0x7ff09c06f1c0 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 >> v2:172.21.15.121:3300/0 conn(0x7ff09c146840 msgr2=0x7ff09c0b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 --2- 172.21.15.46:0/3702055604 >> v2:172.21.15.121:3300/0 conn(0x7ff09c146840 0x7ff09c0b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 >> v1:172.21.15.121:6789/0 conn(0x7ff09c146460 legacy=0x7ff09c06eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c06c740 con 0x7ff09c06f1c0 2023-12-16T21:14:09.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7ff084001670 con 0x7ff09c06f1c0 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 >> v1:172.21.15.46:6789/0 conn(0x7ff09c06f1c0 legacy=0x7ff09c15db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff088003970 0x7ff088003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0880047c0 0x7ff088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff088005180 0x7ff08800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c06c460 con 0x7ff088003970 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08800c020 con 0x7ff088005180 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08800c1e0 con 0x7ff0880047c0 2023-12-16T21:14:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09affd640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff088003970 0x7ff088003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09b7fe640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff088005180 0x7ff08800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09a7fc640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0880047c0 0x7ff088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09affd640 1 -- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0880047c0 msgr2=0x7ff088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09affd640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0880047c0 0x7ff088004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09affd640 1 -- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff088005180 msgr2=0x7ff08800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09affd640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff088005180 0x7ff08800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09affd640 1 -- 172.21.15.46:0/3702055604 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088003520 con 0x7ff088003970 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09b7fe640 1 -- 172.21.15.46:0/3702055604 reap_dead start 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.900+0000 7ff09a7fc640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0880047c0 0x7ff088004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.901+0000 7ff09affd640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff088003970 0x7ff088003d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff094001750 tx=0x7ff0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbef1335c9f34fa3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.901+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0940052d0 con 0x7ff088003970 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.901+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09402e050 con 0x7ff088003970 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.901+0000 7ff099ffb640 1 -- 172.21.15.46:0/3702055604 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094003830 con 0x7ff088003970 2023-12-16T21:14:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 -- 172.21.15.46:0/3702055604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff088003970 msgr2=0x7ff088003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff088003970 0x7ff088003d90 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff094001750 tx=0x7ff0880057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 -- 172.21.15.46:0/3702055604 shutdown_connections 2023-12-16T21:14:09.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 --2- 172.21.15.46:0/3702055604 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff088003970 0x7ff088003d90 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 -- 172.21.15.46:0/3702055604 >> 172.21.15.46:0/3702055604 conn(0x7ff09c06a6b0 msgr2=0x7ff09c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:09.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 -- 172.21.15.46:0/3702055604 shutdown_connections 2023-12-16T21:14:09.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.902+0000 7ff0a1790640 1 -- 172.21.15.46:0/3702055604 wait complete. 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.903+0000 7ff0a1790640 1 Processor -- start 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.903+0000 7ff0a1790640 1 -- start start 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff0a1790640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff088003970 0x7ff09c0d1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff0a1790640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09c0cc710 0x7ff09c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff0a1790640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 0x7ff09c0ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff0a1790640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0d1660 con 0x7ff09c0ca370 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff0a1790640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0cda90 con 0x7ff09c0cc710 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff0a1790640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0cce80 con 0x7ff088003970 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 0x7ff09c0ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 0x7ff09c0ca7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57842/0 (socket says 172.21.15.46:57842) 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 -- 172.21.15.46:0/1065393856 learned_addr learned my addr 172.21.15.46:0/1065393856 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:09.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09affd640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff088003970 0x7ff09c0d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09a7fc640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09c0cc710 0x7ff09c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 -- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff088003970 msgr2=0x7ff09c0d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff088003970 0x7ff09c0d1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 -- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09c0cc710 msgr2=0x7ff09c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09c0cc710 0x7ff09c15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 -- 172.21.15.46:0/1065393856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c0cd160 con 0x7ff09c0ca370 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.904+0000 7ff09b7fe640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 0x7ff09c0ca7f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0b5e10 tx=0x7ff08c011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.905+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00e040 con 0x7ff09c0ca370 2023-12-16T21:14:09.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.905+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c00f040 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.905+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09c0cd440 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.905+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c00feb0 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.905+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09c0ccb90 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.906+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09c069400 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.906+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff08c015430 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.906+0000 7ff0937fe640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07c06ae40 0x7ff07c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.907+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff08c08a050 con 0x7ff09c0ca370 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.907+0000 7ff09affd640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07c06ae40 0x7ff07c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:09.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.907+0000 7ff09affd640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07c06ae40 0x7ff07c06d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff094033000 tx=0x7ff094009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:09.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:09.911+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff08c055100 con 0x7ff09c0ca370 2023-12-16T21:14:10.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.087+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff09c0cdbd0 con 0x7ff09c0ca370 2023-12-16T21:14:10.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.088+0000 7ff0937fe640 1 -- 172.21.15.46:0/1065393856 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff08c058720 con 0x7ff09c0ca370 2023-12-16T21:14:10.088 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07c06ae40 msgr2=0x7ff07c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07c06ae40 0x7ff07c06d300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff094033000 tx=0x7ff094009030 comp rx=0 tx=0).stop 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 msgr2=0x7ff09c0ca7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 0x7ff09c0ca7f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0b5e10 tx=0x7ff08c011f20 comp rx=0 tx=0).stop 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 shutdown_connections 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07c06ae40 0x7ff07c06d300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff09c0ca370 0x7ff09c0ca7f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09c0cc710 0x7ff09c15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 --2- 172.21.15.46:0/1065393856 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff088003970 0x7ff09c0d1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 >> 172.21.15.46:0/1065393856 conn(0x7ff09c06a6b0 msgr2=0x7ff09c0b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 shutdown_connections 2023-12-16T21:14:10.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.090+0000 7ff0a1790640 1 -- 172.21.15.46:0/1065393856 wait complete. 2023-12-16T21:14:10.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:10.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.210+0000 7f22442c3640 1 Processor -- start 2023-12-16T21:14:10.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.210+0000 7f22442c3640 1 -- start start 2023-12-16T21:14:10.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.211+0000 7f22442c3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f223c15d2e0 0x7f223c0d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.211+0000 7f22442c3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f223c06bbd0 0x7f223c0d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.211+0000 7f22442c3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c06ce00 con 0x7f223c15d6c0 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.211+0000 7f22442c3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223c06c1f0 con 0x7f223c06bbd0 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.211+0000 7f22442c3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f223c06c4d0 con 0x7f223c15d2e0 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.211+0000 7f2242860640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f223c06bbd0 0x7f223c0d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f2242860640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f223c06bbd0 0x7f223c0d1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57846/0 (socket says 172.21.15.46:57846) 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f2242860640 1 -- 172.21.15.46:0/3757380495 learned_addr learned my addr 172.21.15.46:0/3757380495 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f224205f640 1 --2- 172.21.15.46:0/3757380495 >> v2:172.21.15.121:3300/0 conn(0x7f223c15d2e0 0x7f223c0d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f2242860640 1 -- 172.21.15.46:0/3757380495 >> v2:172.21.15.121:3300/0 conn(0x7f223c15d2e0 msgr2=0x7f223c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f2242860640 1 --2- 172.21.15.46:0/3757380495 >> v2:172.21.15.121:3300/0 conn(0x7f223c15d2e0 0x7f223c0d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f2242860640 1 -- 172.21.15.46:0/3757380495 >> v1:172.21.15.121:6789/0 conn(0x7f223c15d6c0 legacy=0x7f223c06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.212+0000 7f2242860640 1 -- 172.21.15.46:0/3757380495 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c06c7b0 con 0x7f223c06bbd0 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f2242860640 1 --2- 172.21.15.46:0/3757380495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c06bbd0 0x7f223c0d1b80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f223c08ccd0 tx=0x7f222c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e966ab1101575e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f224105d640 1 -- 172.21.15.46:0/3757380495 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c02ca80 con 0x7f223c06bbd0 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f224105d640 1 -- 172.21.15.46:0/3757380495 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c02cbe0 con 0x7f223c06bbd0 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f224105d640 1 -- 172.21.15.46:0/3757380495 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c0343f0 con 0x7f223c06bbd0 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 -- 172.21.15.46:0/3757380495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c06bbd0 msgr2=0x7f223c0d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 --2- 172.21.15.46:0/3757380495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c06bbd0 0x7f223c0d1b80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f223c08ccd0 tx=0x7f222c02bee0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 -- 172.21.15.46:0/3757380495 shutdown_connections 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 --2- 172.21.15.46:0/3757380495 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c06bbd0 0x7f223c0d1b80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 --2- 172.21.15.46:0/3757380495 >> v2:172.21.15.121:3300/0 conn(0x7f223c15d2e0 0x7f223c0d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 -- 172.21.15.46:0/3757380495 >> 172.21.15.46:0/3757380495 conn(0x7f223c09c580 msgr2=0x7f223c0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:10.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 -- 172.21.15.46:0/3757380495 shutdown_connections 2023-12-16T21:14:10.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.213+0000 7f22442c3640 1 -- 172.21.15.46:0/3757380495 wait complete. 2023-12-16T21:14:10.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 Processor -- start 2023-12-16T21:14:10.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 -- start start 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f223c06bbd0 0x7f223c0b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f223c15d2e0 0x7f223c0cf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 0x7f223c15fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0b6160 con 0x7f223c15d6c0 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0b8c80 con 0x7f223c15d2e0 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f22442c3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0b4460 con 0x7f223c06bbd0 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f2242860640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 0x7f223c15fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f2242860640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 0x7f223c15fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57856/0 (socket says 172.21.15.46:57856) 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.215+0000 7f2242860640 1 -- 172.21.15.46:0/2672696264 learned_addr learned my addr 172.21.15.46:0/2672696264 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f224185e640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f223c15d2e0 0x7f223c0cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f224205f640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f223c06bbd0 0x7f223c0b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f2242860640 1 -- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f223c06bbd0 msgr2=0x7f223c0b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f2242860640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f223c06bbd0 0x7f223c0b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f2242860640 1 -- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f223c15d2e0 msgr2=0x7f223c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f2242860640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f223c15d2e0 0x7f223c0cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f2242860640 1 -- 172.21.15.46:0/2672696264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c009b30 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f224205f640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f223c06bbd0 0x7f223c0b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f2242860640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 0x7f223c15fe50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f222c036040 tx=0x7f222c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c009510 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f222c03e450 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c0b8100 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c03e800 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.216+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c0b86c0 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.217+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223c067420 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.218+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f222c02cd50 con 0x7f223c15d6c0 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.218+0000 7f223affd640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f222806ae40 0x7f222806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.218+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f222c0ac060 con 0x7f223c15d6c0 2023-12-16T21:14:10.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.218+0000 7f224205f640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f222806ae40 0x7f222806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.219+0000 7f224205f640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f222806ae40 0x7f222806d300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2234006fd0 tx=0x7f2234008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.223+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f222c077110 con 0x7f223c15d6c0 2023-12-16T21:14:10.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.398+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f223c0d1ad0 con 0x7f223c15d6c0 2023-12-16T21:14:10.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.398+0000 7f223affd640 1 -- 172.21.15.46:0/2672696264 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f222c07a730 con 0x7f223c15d6c0 2023-12-16T21:14:10.398 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:10.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f222806ae40 msgr2=0x7f222806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f222806ae40 0x7f222806d300 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2234006fd0 tx=0x7f2234008040 comp rx=0 tx=0).stop 2023-12-16T21:14:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 msgr2=0x7f223c15fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 0x7f223c15fe50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f222c036040 tx=0x7f222c004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 shutdown_connections 2023-12-16T21:14:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f222806ae40 0x7f222806d300 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f223c15d6c0 0x7f223c15fe50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f223c15d2e0 0x7f223c0cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 --2- 172.21.15.46:0/2672696264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f223c06bbd0 0x7f223c0b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 >> 172.21.15.46:0/2672696264 conn(0x7f223c09c580 msgr2=0x7f223c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:10.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 shutdown_connections 2023-12-16T21:14:10.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.400+0000 7f22442c3640 1 -- 172.21.15.46:0/2672696264 wait complete. 2023-12-16T21:14:10.412 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:10.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.520+0000 7f87ed7a9640 1 Processor -- start 2023-12-16T21:14:10.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.520+0000 7f87ed7a9640 1 -- start start 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87ed7a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87e806ea50 0x7f87e806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87ed7a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e80d0ed0 0x7f87e80d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87ed7a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87e8161d20 con 0x7f87e8160fb0 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87ed7a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e806f370 con 0x7f87e806ea50 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87ed7a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e815e760 con 0x7f87e80d0ed0 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87e806ea50 0x7f87e806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f87e806ea50 0x7f87e806ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57870/0 (socket says 172.21.15.46:57870) 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 -- 172.21.15.46:0/1848070760 learned_addr learned my addr 172.21.15.46:0/1848070760 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e77fe640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e80d0ed0 0x7f87e80d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 -- 172.21.15.46:0/1848070760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e80d0ed0 msgr2=0x7f87e80d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e80d0ed0 0x7f87e80d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 -- 172.21.15.46:0/1848070760 >> v1:172.21.15.121:6789/0 conn(0x7f87e8160fb0 legacy=0x7f87e8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e6ffd640 1 -- 172.21.15.46:0/1848070760 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e815d9a0 con 0x7f87e806ea50 2023-12-16T21:14:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.521+0000 7f87e77fe640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e80d0ed0 0x7f87e80d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.522+0000 7f87e6ffd640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e806ee30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009870 tx=0x7f87e002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456209d8b961efd9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.522+0000 7f87e5ffb640 1 -- 172.21.15.46:0/1848070760 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0030840 con 0x7f87e806ea50 2023-12-16T21:14:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.522+0000 7f87e5ffb640 1 -- 172.21.15.46:0/1848070760 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e00309a0 con 0x7f87e806ea50 2023-12-16T21:14:10.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.522+0000 7f87e5ffb640 1 -- 172.21.15.46:0/1848070760 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0030d30 con 0x7f87e806ea50 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 -- 172.21.15.46:0/1848070760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 msgr2=0x7f87e806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e806ee30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009870 tx=0x7f87e002b980 comp rx=0 tx=0).stop 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 -- 172.21.15.46:0/1848070760 shutdown_connections 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e80d0ed0 0x7f87e80d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/1848070760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e806ee30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 -- 172.21.15.46:0/1848070760 >> 172.21.15.46:0/1848070760 conn(0x7f87e806a6b0 msgr2=0x7f87e806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 -- 172.21.15.46:0/1848070760 shutdown_connections 2023-12-16T21:14:10.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.523+0000 7f87ed7a9640 1 -- 172.21.15.46:0/1848070760 wait complete. 2023-12-16T21:14:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.524+0000 7f87ed7a9640 1 Processor -- start 2023-12-16T21:14:10.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.524+0000 7f87ed7a9640 1 -- start start 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87ed7a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e8157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87ed7a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87e80d0ed0 0x7f87e81e6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87ed7a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e8160fb0 0x7f87e81e73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87ed7a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e813f500 con 0x7f87e806ea50 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87ed7a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e81588a0 con 0x7f87e80d0ed0 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87ed7a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e8158b80 con 0x7f87e8160fb0 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e8157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e8157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57884/0 (socket says 172.21.15.46:57884) 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 -- 172.21.15.46:0/3827253859 learned_addr learned my addr 172.21.15.46:0/3827253859 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:10.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e67fc640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87e80d0ed0 0x7f87e81e6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e77fe640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e8160fb0 0x7f87e81e73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 -- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e8160fb0 msgr2=0x7f87e81e73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e8160fb0 0x7f87e81e73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 -- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87e80d0ed0 msgr2=0x7f87e81e6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.525+0000 7f87e6ffd640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87e80d0ed0 0x7f87e81e6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87e6ffd640 1 -- 172.21.15.46:0/3827253859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e00094d0 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87e6ffd640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e8157ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f87e0000c00 tx=0x7f87e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0030810 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e0039d90 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e8158e60 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e0043670 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.526+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e8159420 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.527+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e8159610 con 0x7f87e806ea50 2023-12-16T21:14:10.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.527+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f87e004c3b0 con 0x7f87e806ea50 2023-12-16T21:14:10.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.528+0000 7f87df7fe640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f87c806ae40 0x7f87c806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.528+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87e00abc20 con 0x7f87e806ea50 2023-12-16T21:14:10.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.528+0000 7f87e67fc640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f87c806ae40 0x7f87c806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.529+0000 7f87e67fc640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f87c806ae40 0x7f87c806d300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87e81e9430 tx=0x7f87d0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.532+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f87e0076d60 con 0x7f87e806ea50 2023-12-16T21:14:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.722+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f87e8069400 con 0x7f87e806ea50 2023-12-16T21:14:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.723+0000 7f87df7fe640 1 -- 172.21.15.46:0/3827253859 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f87e007a380 con 0x7f87e806ea50 2023-12-16T21:14:10.722 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f87c806ae40 msgr2=0x7f87c806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f87c806ae40 0x7f87c806d300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87e81e9430 tx=0x7f87d0009040 comp rx=0 tx=0).stop 2023-12-16T21:14:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 msgr2=0x7f87e8157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e8157ec0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f87e0000c00 tx=0x7f87e0004730 comp rx=0 tx=0).stop 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 shutdown_connections 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f87c806ae40 0x7f87c806d300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87e8160fb0 0x7f87e81e73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87e80d0ed0 0x7f87e81e6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 --2- 172.21.15.46:0/3827253859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87e806ea50 0x7f87e8157ec0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 >> 172.21.15.46:0/3827253859 conn(0x7f87e806a6b0 msgr2=0x7f87e80b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:10.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 shutdown_connections 2023-12-16T21:14:10.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.725+0000 7f87ed7a9640 1 -- 172.21.15.46:0/3827253859 wait complete. 2023-12-16T21:14:10.735 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:10.736 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:10.753 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:10.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.900+0000 7f3eeacc9640 1 Processor -- start 2023-12-16T21:14:10.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.900+0000 7f3eeacc9640 1 -- start start 2023-12-16T21:14:10.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.901+0000 7f3eeacc9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3ee4160bb0 0x7f3ee40d3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.901+0000 7f3eeacc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40b3450 con 0x7f3ee40ce440 2023-12-16T21:14:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.901+0000 7f3eeacc9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee40b45b0 con 0x7f3ee40b3fc0 2023-12-16T21:14:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.901+0000 7f3eeacc9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee40b4890 con 0x7f3ee4160bb0 2023-12-16T21:14:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.901+0000 7f3ee9266640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3ee40ce440 0x7f3ee40d1120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33424/0 (socket says 172.21.15.46:33424) 2023-12-16T21:14:10.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.901+0000 7f3ee9266640 1 -- 172.21.15.46:0/2057466148 learned_addr learned my addr 172.21.15.46:0/2057466148 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee8a65640 1 --2- 172.21.15.46:0/2057466148 >> v2:172.21.15.121:3300/0 conn(0x7f3ee4160bb0 0x7f3ee40d3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794397823 0 0) 0x7f3ee40b3450 con 0x7f3ee40ce440 2023-12-16T21:14:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ecc003520 con 0x7f3ee40ce440 2023-12-16T21:14:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3777765843 0 0) 0x7f3ecc003520 con 0x7f3ee40ce440 2023-12-16T21:14:10.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 >> v2:172.21.15.121:3300/0 conn(0x7f3ee4160bb0 msgr2=0x7f3ee40d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 --2- 172.21.15.46:0/2057466148 >> v2:172.21.15.121:3300/0 conn(0x7f3ee4160bb0 0x7f3ee40d3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 >> v1:172.21.15.121:6789/0 conn(0x7f3ee40b3fc0 legacy=0x7f3ee40cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40b36a0 con 0x7f3ee40ce440 2023-12-16T21:14:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f3edc0019f0 con 0x7f3ee40ce440 2023-12-16T21:14:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 >> v1:172.21.15.46:6789/0 conn(0x7f3ee40ce440 legacy=0x7f3ee40d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc003970 0x7f3ecc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ecc0047c0 0x7f3ecc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ecc005180 0x7f3ecc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b4890 con 0x7f3ecc005180 2023-12-16T21:14:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc00c020 con 0x7f3ecc0047c0 2023-12-16T21:14:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ecc00c1e0 con 0x7f3ecc003970 2023-12-16T21:14:10.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee9266640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ecc005180 0x7f3ecc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee8a65640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc003970 0x7f3ecc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee3fff640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ecc0047c0 0x7f3ecc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee9266640 1 -- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc003970 msgr2=0x7f3ecc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee9266640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc003970 0x7f3ecc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee9266640 1 -- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ecc0047c0 msgr2=0x7f3ecc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee9266640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ecc0047c0 0x7f3ecc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.902+0000 7f3ee9266640 1 -- 172.21.15.46:0/2057466148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc003520 con 0x7f3ecc005180 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.903+0000 7f3ee9266640 1 -- 172.21.15.46:0/2057466148 reap_dead start 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.903+0000 7f3ee9266640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ecc005180 0x7f3ecc00b980 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00a830 tx=0x7f3ecc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9680a13753b06677 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.903+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc0072d0 con 0x7f3ecc005180 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.903+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc02f070 con 0x7f3ecc005180 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.903+0000 7f3ee37fe640 1 -- 172.21.15.46:0/2057466148 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc0076b0 con 0x7f3ecc005180 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 -- 172.21.15.46:0/2057466148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ecc005180 msgr2=0x7f3ecc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ecc005180 0x7f3ecc00b980 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00a830 tx=0x7f3ecc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 -- 172.21.15.46:0/2057466148 shutdown_connections 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/2057466148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ecc005180 0x7f3ecc00b980 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 -- 172.21.15.46:0/2057466148 >> 172.21.15.46:0/2057466148 conn(0x7f3ee409c590 msgr2=0x7f3ee40d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 -- 172.21.15.46:0/2057466148 shutdown_connections 2023-12-16T21:14:10.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.904+0000 7f3eeacc9640 1 -- 172.21.15.46:0/2057466148 wait complete. 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.905+0000 7f3eeacc9640 1 Processor -- start 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.905+0000 7f3eeacc9640 1 -- start start 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3eeacc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc005180 0x7f3ee40b7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3eeacc9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee415efb0 0x7f3ee4154050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3eeacc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 0x7f3ee40cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3eeacc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40d0470 con 0x7f3ee40cfad0 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3eeacc9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee415ae50 con 0x7f3ee415efb0 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3eeacc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee40b8660 con 0x7f3ecc005180 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 0x7f3ee40cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 0x7f3ee40cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53158/0 (socket says 172.21.15.46:53158) 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 -- 172.21.15.46:0/233776567 learned_addr learned my addr 172.21.15.46:0/233776567 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:10.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee8a65640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc005180 0x7f3ee40b7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee3fff640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee415efb0 0x7f3ee4154050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 -- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc005180 msgr2=0x7f3ee40b7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc005180 0x7f3ee40b7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 -- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee415efb0 msgr2=0x7f3ee4154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee415efb0 0x7f3ee4154050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 -- 172.21.15.46:0/233776567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee40b80a0 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.906+0000 7f3ee9266640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 0x7f3ee40cff30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00a920 tx=0x7f3edc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.907+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc007650 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.907+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc0077b0 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.907+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee40b8380 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.907+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3edc0019c0 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.907+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee415f430 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.908+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee415b000 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.908+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3edc031030 con 0x7f3ee40cfad0 2023-12-16T21:14:10.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.908+0000 7f3ee17fa640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ed006ae40 0x7f3ed006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.909+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3edc0ab110 con 0x7f3ee40cfad0 2023-12-16T21:14:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.909+0000 7f3ee8a65640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ed006ae40 0x7f3ed006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.909+0000 7f3ee8a65640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ed006ae40 0x7f3ed006d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009980 tx=0x7f3ed4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:10.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:10.913+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3ee415b000 con 0x7f3ee40cfad0 2023-12-16T21:14:11.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.090+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ee415b000 con 0x7f3ee40cfad0 2023-12-16T21:14:11.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.091+0000 7f3ee17fa640 1 -- 172.21.15.46:0/233776567 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f3ee415b000 con 0x7f3ee40cfad0 2023-12-16T21:14:11.091 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:11.091 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:11.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ed006ae40 msgr2=0x7f3ed006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ed006ae40 0x7f3ed006d300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4009980 tx=0x7f3ed4008040 comp rx=0 tx=0).stop 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 msgr2=0x7f3ee40cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 0x7f3ee40cff30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00a920 tx=0x7f3edc001d80 comp rx=0 tx=0).stop 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 shutdown_connections 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3ed006ae40 0x7f3ed006d300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3ee40cfad0 0x7f3ee40cff30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ee415efb0 0x7f3ee4154050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 --2- 172.21.15.46:0/233776567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ecc005180 0x7f3ee40b7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 >> 172.21.15.46:0/233776567 conn(0x7f3ee409c590 msgr2=0x7f3ee406bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 shutdown_connections 2023-12-16T21:14:11.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.093+0000 7f3eeacc9640 1 -- 172.21.15.46:0/233776567 wait complete. 2023-12-16T21:14:11.103 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:11.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:11.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.214+0000 7f0885316640 1 Processor -- start 2023-12-16T21:14:11.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.214+0000 7f0885316640 1 -- start start 2023-12-16T21:14:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.214+0000 7f0885316640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f08800b4370 0x7f08800b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.214+0000 7f0885316640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0880071770 con 0x7f0880144980 2023-12-16T21:14:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.214+0000 7f0885316640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08800ccb60 con 0x7f08800ceb70 2023-12-16T21:14:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.214+0000 7f0885316640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f088015ced0 con 0x7f08800b4370 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0880144980 0x7f088015c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33434/0 (socket says 172.21.15.46:33434) 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087effd640 1 -- 172.21.15.46:0/3028222673 learned_addr learned my addr 172.21.15.46:0/3028222673 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087f7fe640 1 --2- 172.21.15.46:0/3028222673 >> v2:172.21.15.121:3300/0 conn(0x7f08800b4370 0x7f08800b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834993289 0 0) 0x7f0880071770 con 0x7f0880144980 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0868003520 con 0x7f0880144980 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716223612 0 0) 0x7f08800ccb60 con 0x7f08800ceb70 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0880071770 con 0x7f08800ceb70 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087f7fe640 1 -- 172.21.15.46:0/3028222673 >> v1:172.21.15.121:6789/0 conn(0x7f08800ceb70 legacy=0x7f08800cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087f7fe640 1 -- 172.21.15.46:0/3028222673 >> v1:172.21.15.46:6789/0 conn(0x7f0880144980 legacy=0x7f088015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.215+0000 7f087f7fe640 1 -- 172.21.15.46:0/3028222673 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08800b4ce0 con 0x7f08800b4370 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.216+0000 7f087f7fe640 1 --2- 172.21.15.46:0/3028222673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08800b4370 0x7f08800b47a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f08800b4e70 tx=0x7f087802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25c522afcbbc7f45 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.216+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f087802cce0 con 0x7f08800b4370 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.216+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087802ce40 con 0x7f08800b4370 2023-12-16T21:14:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.216+0000 7f087dffb640 1 -- 172.21.15.46:0/3028222673 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0878034660 con 0x7f08800b4370 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.216+0000 7f0885316640 1 -- 172.21.15.46:0/3028222673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08800b4370 msgr2=0x7f08800b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.216+0000 7f0885316640 1 --2- 172.21.15.46:0/3028222673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08800b4370 0x7f08800b47a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f08800b4e70 tx=0x7f087802bf30 comp rx=0 tx=0).stop 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.217+0000 7f0885316640 1 -- 172.21.15.46:0/3028222673 shutdown_connections 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.217+0000 7f0885316640 1 --2- 172.21.15.46:0/3028222673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08800b4370 0x7f08800b47a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.217+0000 7f0885316640 1 -- 172.21.15.46:0/3028222673 >> 172.21.15.46:0/3028222673 conn(0x7f088006a4d0 msgr2=0x7f088006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.217+0000 7f0885316640 1 -- 172.21.15.46:0/3028222673 shutdown_connections 2023-12-16T21:14:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.217+0000 7f0885316640 1 -- 172.21.15.46:0/3028222673 wait complete. 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.218+0000 7f0885316640 1 Processor -- start 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.218+0000 7f0885316640 1 -- start start 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 0x7f088013b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08800ceb70 0x7f0880157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0880144980 0x7f0880158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088013b9e0 con 0x7f08800b4370 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08800b29e0 con 0x7f0880144980 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f088013bee0 con 0x7f08800ceb70 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 0x7f088013b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 0x7f088013b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53168/0 (socket says 172.21.15.46:53168) 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 -- 172.21.15.46:0/903628043 learned_addr learned my addr 172.21.15.46:0/903628043 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087f7fe640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0880144980 0x7f0880158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087e7fc640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08800ceb70 0x7f0880157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 -- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08800ceb70 msgr2=0x7f0880157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08800ceb70 0x7f0880157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 -- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0880144980 msgr2=0x7f0880158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0880144980 0x7f0880158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 -- 172.21.15.46:0/903628043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878009a70 con 0x7f08800b4370 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087f7fe640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0880144980 0x7f0880158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087e7fc640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08800ceb70 0x7f0880157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f087effd640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 0x7f088013b4a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f087000c430 tx=0x7f087000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870005ce0 con 0x7f08800b4370 2023-12-16T21:14:11.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0870005e40 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.219+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08801f69e0 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.220+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08700072f0 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.220+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08800b2390 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.221+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0880069400 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.221+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0870007450 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.221+0000 7f08677fe640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f086c06ae40 0x7f086c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.221+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0870088c70 con 0x7f08800b4370 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.221+0000 7f087e7fc640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f086c06ae40 0x7f086c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.222+0000 7f087e7fc640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f086c06ae40 0x7f086c06d300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f08800cb7e0 tx=0x7f0874009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.225+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0870053d20 con 0x7f08800b4370 2023-12-16T21:14:11.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.401+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f088006b4e0 con 0x7f08800b4370 2023-12-16T21:14:11.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.402+0000 7f08677fe640 1 -- 172.21.15.46:0/903628043 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f08700170a0 con 0x7f08800b4370 2023-12-16T21:14:11.402 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f086c06ae40 msgr2=0x7f086c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f086c06ae40 0x7f086c06d300 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f08800cb7e0 tx=0x7f0874009040 comp rx=0 tx=0).stop 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 msgr2=0x7f088013b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 0x7f088013b4a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f087000c430 tx=0x7f087000c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 shutdown_connections 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f086c06ae40 0x7f086c06d300 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0880144980 0x7f0880158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08800ceb70 0x7f0880157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 --2- 172.21.15.46:0/903628043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08800b4370 0x7f088013b4a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 >> 172.21.15.46:0/903628043 conn(0x7f088006a4d0 msgr2=0x7f0880156700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 shutdown_connections 2023-12-16T21:14:11.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.404+0000 7f0885316640 1 -- 172.21.15.46:0/903628043 wait complete. 2023-12-16T21:14:11.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:11.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.524+0000 7efc6737c640 1 Processor -- start 2023-12-16T21:14:11.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.524+0000 7efc6737c640 1 -- start start 2023-12-16T21:14:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc6737c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc60150940 0x7efc6015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc6737c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efc6015dec0 0x7efc6015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc6737c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b49f0 con 0x7efc60150560 2023-12-16T21:14:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc6737c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc600b3560 con 0x7efc6015dec0 2023-12-16T21:14:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc6737c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc600b3de0 con 0x7efc60150940 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc65919640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efc6015dec0 0x7efc6015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc65919640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efc6015dec0 0x7efc6015d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53176/0 (socket says 172.21.15.46:53176) 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc65919640 1 -- 172.21.15.46:0/377140808 learned_addr learned my addr 172.21.15.46:0/377140808 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.525+0000 7efc64917640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc60150940 0x7efc6015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc65919640 1 -- 172.21.15.46:0/377140808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc60150940 msgr2=0x7efc6015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc65919640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc60150940 0x7efc6015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc65919640 1 -- 172.21.15.46:0/377140808 >> v1:172.21.15.121:6789/0 conn(0x7efc60150560 legacy=0x7efc600b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc65919640 1 -- 172.21.15.46:0/377140808 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b40c0 con 0x7efc6015dec0 2023-12-16T21:14:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc64917640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc60150940 0x7efc6015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc65919640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc6015dec0 0x7efc6015d740 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efc50009e50 tx=0x7efc5002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b0ef003d74658c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.526+0000 7efc5ffff640 1 -- 172.21.15.46:0/377140808 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5002cb00 con 0x7efc6015dec0 2023-12-16T21:14:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc5ffff640 1 -- 172.21.15.46:0/377140808 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5002cc60 con 0x7efc6015dec0 2023-12-16T21:14:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc5ffff640 1 -- 172.21.15.46:0/377140808 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500343f0 con 0x7efc6015dec0 2023-12-16T21:14:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 -- 172.21.15.46:0/377140808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc6015dec0 msgr2=0x7efc6015d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc6015dec0 0x7efc6015d740 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efc50009e50 tx=0x7efc5002bf60 comp rx=0 tx=0).stop 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 -- 172.21.15.46:0/377140808 shutdown_connections 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc60150940 0x7efc6015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 --2- 172.21.15.46:0/377140808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc6015dec0 0x7efc6015d740 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 -- 172.21.15.46:0/377140808 >> 172.21.15.46:0/377140808 conn(0x7efc6006a4d0 msgr2=0x7efc6006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 -- 172.21.15.46:0/377140808 shutdown_connections 2023-12-16T21:14:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.527+0000 7efc6737c640 1 -- 172.21.15.46:0/377140808 wait complete. 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.528+0000 7efc6737c640 1 Processor -- start 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.528+0000 7efc6737c640 1 -- start start 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc6737c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 0x7efc60145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc6737c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc6015dec0 0x7efc6014cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc6737c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc6014d090 0x7efc6015bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc6737c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6015ae50 con 0x7efc60150560 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc6737c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6015b130 con 0x7efc6015dec0 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc6737c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6015b410 con 0x7efc6014d090 2023-12-16T21:14:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc65118640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 0x7efc60145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc65118640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 0x7efc60145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53182/0 (socket says 172.21.15.46:53182) 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc65118640 1 -- 172.21.15.46:0/1848524066 learned_addr learned my addr 172.21.15.46:0/1848524066 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc64917640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc6015dec0 0x7efc6014cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.529+0000 7efc65919640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc6014d090 0x7efc6015bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc65118640 1 -- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc6014d090 msgr2=0x7efc6015bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc65118640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc6014d090 0x7efc6015bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc65118640 1 -- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc6015dec0 msgr2=0x7efc6014cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc65118640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc6015dec0 0x7efc6014cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc65118640 1 -- 172.21.15.46:0/1848524066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50009b30 con 0x7efc60150560 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc65118640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 0x7efc60145130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efc5800c450 tx=0x7efc5800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc580052d0 con 0x7efc60150560 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc58005430 con 0x7efc60150560 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc58012870 con 0x7efc60150560 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc6015b6f0 con 0x7efc60150560 2023-12-16T21:14:11.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.530+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc60149a90 con 0x7efc60150560 2023-12-16T21:14:11.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.531+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc60069400 con 0x7efc60150560 2023-12-16T21:14:11.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.532+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7efc580129d0 con 0x7efc60150560 2023-12-16T21:14:11.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.532+0000 7efc5dffb640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efc4c06ae40 0x7efc4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.532+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc580890c0 con 0x7efc60150560 2023-12-16T21:14:11.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.532+0000 7efc64917640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efc4c06ae40 0x7efc4c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.533+0000 7efc64917640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efc4c06ae40 0x7efc4c06d300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc60071430 tx=0x7efc54008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.536+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7efc58054200 con 0x7efc60150560 2023-12-16T21:14:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.714+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc6015af90 con 0x7efc60150560 2023-12-16T21:14:11.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.714+0000 7efc5dffb640 1 -- 172.21.15.46:0/1848524066 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc6015af90 con 0x7efc60150560 2023-12-16T21:14:11.714 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:11.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.716+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efc4c06ae40 msgr2=0x7efc4c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.716+0000 7efc6737c640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efc4c06ae40 0x7efc4c06d300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efc60071430 tx=0x7efc54008040 comp rx=0 tx=0).stop 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.716+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 msgr2=0x7efc60145130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.716+0000 7efc6737c640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 0x7efc60145130 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efc5800c450 tx=0x7efc5800c910 comp rx=0 tx=0).stop 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 shutdown_connections 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efc4c06ae40 0x7efc4c06d300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc6014d090 0x7efc6015bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc6015dec0 0x7efc6014cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 --2- 172.21.15.46:0/1848524066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efc60150560 0x7efc60145130 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 >> 172.21.15.46:0/1848524066 conn(0x7efc6006a4d0 msgr2=0x7efc6015d430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 shutdown_connections 2023-12-16T21:14:11.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.717+0000 7efc6737c640 1 -- 172.21.15.46:0/1848524066 wait complete. 2023-12-16T21:14:11.727 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:11.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.837+0000 7f68f85d7640 1 Processor -- start 2023-12-16T21:14:11.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.837+0000 7f68f85d7640 1 -- start start 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f85d7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f85d7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f0161d20 con 0x7f68f0160fb0 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f85d7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68f015e760 con 0x7f68f006ea50 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f85d7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f00d1ee0 con 0x7f68f0142270 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5b72640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f68f0160fb0 0x7f68f0161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33446/0 (socket says 172.21.15.46:33446) 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5b72640 1 -- 172.21.15.46:0/3567674100 learned_addr learned my addr 172.21.15.46:0/3567674100 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f6b74640 1 --2- 172.21.15.46:0/3567674100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934628447 0 0) 0x7f68f0161d20 con 0x7f68f0160fb0 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68dc003520 con 0x7f68f0160fb0 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910851980 0 0) 0x7f68f015e760 con 0x7f68f006ea50 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f0161d20 con 0x7f68f006ea50 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3625924522 0 0) 0x7f68dc003520 con 0x7f68f0160fb0 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f6b74640 1 -- 172.21.15.46:0/3567674100 >> v1:172.21.15.121:6789/0 conn(0x7f68f006ea50 legacy=0x7f68f006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f6b74640 1 -- 172.21.15.46:0/3567674100 >> v1:172.21.15.46:6789/0 conn(0x7f68f0160fb0 legacy=0x7f68f0161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.838+0000 7f68f6b74640 1 -- 172.21.15.46:0/3567674100 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f015d9a0 con 0x7f68f0142270 2023-12-16T21:14:11.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.839+0000 7f68f6b74640 1 --2- 172.21.15.46:0/3567674100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01426c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009dc0 tx=0x7f68e802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=450f35873ecadb75 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.839+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0001670 con 0x7f68f0142270 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.839+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e802c910 con 0x7f68f0142270 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f5371640 1 -- 172.21.15.46:0/3567674100 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e802cca0 con 0x7f68f0142270 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 -- 172.21.15.46:0/3567674100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 msgr2=0x7f68f01426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 --2- 172.21.15.46:0/3567674100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01426c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68e8009dc0 tx=0x7f68e802bde0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 -- 172.21.15.46:0/3567674100 shutdown_connections 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 --2- 172.21.15.46:0/3567674100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01426c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 -- 172.21.15.46:0/3567674100 >> 172.21.15.46:0/3567674100 conn(0x7f68f006a6b0 msgr2=0x7f68f006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 -- 172.21.15.46:0/3567674100 shutdown_connections 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.840+0000 7f68f85d7640 1 -- 172.21.15.46:0/3567674100 wait complete. 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.841+0000 7f68f85d7640 1 Processor -- start 2023-12-16T21:14:11.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.841+0000 7f68f85d7640 1 -- start start 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f85d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 0x7f68f006cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f85d7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f85d7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f0160fb0 0x7f68f0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f85d7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f0160700 con 0x7f68f006ea50 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f85d7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f015f5d0 con 0x7f68f0160fb0 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f85d7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f006bb30 con 0x7f68f0142270 2023-12-16T21:14:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 0x7f68f006cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 0x7f68f006cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53190/0 (socket says 172.21.15.46:53190) 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 -- 172.21.15.46:0/1610851823 learned_addr learned my addr 172.21.15.46:0/1610851823 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6b74640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f0160fb0 0x7f68f0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f5b72640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 -- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 msgr2=0x7f68f01601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 -- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f0160fb0 msgr2=0x7f68f0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f0160fb0 0x7f68f0158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.842+0000 7f68f6373640 1 -- 172.21.15.46:0/1610851823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e8009a70 con 0x7f68f006ea50 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.843+0000 7f68f6373640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 0x7f68f006cbd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c430 tx=0x7f68ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.843+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec0052d0 con 0x7f68f006ea50 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.843+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec005450 con 0x7f68f006ea50 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.843+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec003830 con 0x7f68f006ea50 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.843+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68f006be10 con 0x7f68f006ea50 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.843+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68f006c3d0 con 0x7f68f006ea50 2023-12-16T21:14:11.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.844+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0069400 con 0x7f68f006ea50 2023-12-16T21:14:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.844+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f68ec003990 con 0x7f68f006ea50 2023-12-16T21:14:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.845+0000 7f68e6ffd640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68d406ae70 0x7f68d406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:11.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.845+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68ec088b80 con 0x7f68f006ea50 2023-12-16T21:14:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.845+0000 7f68f5b72640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68d406ae70 0x7f68d406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:11.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.845+0000 7f68f5b72640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68d406ae70 0x7f68d406d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68f015b1d0 tx=0x7f68e0001690 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:11.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:11.849+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f68ec053cd0 con 0x7f68f006ea50 2023-12-16T21:14:12.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.025+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f68f015b200 con 0x7f68f006ea50 2023-12-16T21:14:12.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.025+0000 7f68e6ffd640 1 -- 172.21.15.46:0/1610851823 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f68ec088e60 con 0x7f68f006ea50 2023-12-16T21:14:12.025 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68d406ae70 msgr2=0x7f68d406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68d406ae70 0x7f68d406d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f68f015b1d0 tx=0x7f68e0001690 comp rx=0 tx=0).stop 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 msgr2=0x7f68f006cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 0x7f68f006cbd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00c430 tx=0x7f68ec00c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 shutdown_connections 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68d406ae70 0x7f68d406d330 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f0160fb0 0x7f68f0158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f0142270 0x7f68f01601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 --2- 172.21.15.46:0/1610851823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f006ea50 0x7f68f006cbd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 >> 172.21.15.46:0/1610851823 conn(0x7f68f006a6b0 msgr2=0x7f68f00b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:12.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 shutdown_connections 2023-12-16T21:14:12.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.027+0000 7f68f85d7640 1 -- 172.21.15.46:0/1610851823 wait complete. 2023-12-16T21:14:12.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:12.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.152+0000 7fe24f2ad640 1 Processor -- start 2023-12-16T21:14:12.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.152+0000 7fe24f2ad640 1 -- start start 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24f2ad640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe24815d2d0 0x7fe2480b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24f2ad640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2481614e0 con 0x7fe2480ce650 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24f2ad640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2481617c0 con 0x7fe248160f70 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24f2ad640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe248161aa0 con 0x7fe24815d2d0 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d049640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe24815d2d0 0x7fe2480b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d049640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe24815d2d0 0x7fe2480b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53200/0 (socket says 172.21.15.46:53200) 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d049640 1 -- 172.21.15.46:0/3009514715 learned_addr learned my addr 172.21.15.46:0/3009514715 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d84a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2480ce650 0x7fe2480d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33452/0 (socket says 172.21.15.46:33452) 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d049640 1 -- 172.21.15.46:0/3009514715 >> v1:172.21.15.121:6789/0 conn(0x7fe248160f70 legacy=0x7fe2480cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d049640 1 -- 172.21.15.46:0/3009514715 >> v1:172.21.15.46:6789/0 conn(0x7fe2480ce650 legacy=0x7fe2480d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.153+0000 7fe24d049640 1 -- 172.21.15.46:0/3009514715 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe248161d80 con 0x7fe24815d2d0 2023-12-16T21:14:12.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.154+0000 7fe24d049640 1 --2- 172.21.15.46:0/3009514715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe2480b8b90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe244009870 tx=0x7fe24402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4300025262bdb9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.154+0000 7fe23ffff640 1 -- 172.21.15.46:0/3009514715 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24402f840 con 0x7fe24815d2d0 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.154+0000 7fe23ffff640 1 -- 172.21.15.46:0/3009514715 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24402f9a0 con 0x7fe24815d2d0 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.154+0000 7fe23ffff640 1 -- 172.21.15.46:0/3009514715 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24402fd30 con 0x7fe24815d2d0 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 -- 172.21.15.46:0/3009514715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 msgr2=0x7fe2480b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/3009514715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe2480b8b90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe244009870 tx=0x7fe24402ba20 comp rx=0 tx=0).stop 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 -- 172.21.15.46:0/3009514715 shutdown_connections 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/3009514715 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe2480b8b90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 -- 172.21.15.46:0/3009514715 >> 172.21.15.46:0/3009514715 conn(0x7fe248072230 msgr2=0x7fe2480cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 -- 172.21.15.46:0/3009514715 shutdown_connections 2023-12-16T21:14:12.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.155+0000 7fe24f2ad640 1 -- 172.21.15.46:0/3009514715 wait complete. 2023-12-16T21:14:12.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.156+0000 7fe24f2ad640 1 Processor -- start 2023-12-16T21:14:12.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.156+0000 7fe24f2ad640 1 -- start start 2023-12-16T21:14:12.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2480ce650 0x7fe2480cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe24815dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248160f70 0x7fe24815e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480ca2d0 con 0x7fe24815d2d0 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24815e9e0 con 0x7fe248160f70 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24815ecc0 con 0x7fe2480ce650 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe24815dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe24815dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53212/0 (socket says 172.21.15.46:53212) 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 -- 172.21.15.46:0/1313084540 learned_addr learned my addr 172.21.15.46:0/1313084540 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24d84a640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248160f70 0x7fe24815e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24d049640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2480ce650 0x7fe2480cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 -- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2480ce650 msgr2=0x7fe2480cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2480ce650 0x7fe2480cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 -- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248160f70 msgr2=0x7fe24815e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248160f70 0x7fe24815e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 -- 172.21.15.46:0/1313084540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2440094d0 con 0x7fe24815d2d0 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24d84a640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248160f70 0x7fe24815e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24c848640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe24815dc80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe23800c400 tx=0x7fe23800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2380052d0 con 0x7fe24815d2d0 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe238005430 con 0x7fe24815d2d0 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24815efa0 con 0x7fe24815d2d0 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.157+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe238003830 con 0x7fe24815d2d0 2023-12-16T21:14:12.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.158+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24815e700 con 0x7fe24815d2d0 2023-12-16T21:14:12.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.159+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2480ca480 con 0x7fe24815d2d0 2023-12-16T21:14:12.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.159+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe238003990 con 0x7fe24815d2d0 2023-12-16T21:14:12.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.160+0000 7fe23dffb640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe22c06ae40 0x7fe22c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.160+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe238088b50 con 0x7fe24815d2d0 2023-12-16T21:14:12.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.160+0000 7fe24d049640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe22c06ae40 0x7fe22c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.160+0000 7fe24d049640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe22c06ae40 0x7fe22c06d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe244036000 tx=0x7fe244039040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.164+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe2480ca480 con 0x7fe24815d2d0 2023-12-16T21:14:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.353+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe248069400 con 0x7fe24815d2d0 2023-12-16T21:14:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.354+0000 7fe23dffb640 1 -- 172.21.15.46:0/1313084540 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe238053c10 con 0x7fe24815d2d0 2023-12-16T21:14:12.353 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe22c06ae40 msgr2=0x7fe22c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe22c06ae40 0x7fe22c06d300 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe244036000 tx=0x7fe244039040 comp rx=0 tx=0).stop 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 msgr2=0x7fe24815dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe24815dc80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe23800c400 tx=0x7fe23800c8c0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 shutdown_connections 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe22c06ae40 0x7fe22c06d300 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe24815d2d0 0x7fe24815dc80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248160f70 0x7fe24815e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 --2- 172.21.15.46:0/1313084540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2480ce650 0x7fe2480cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 >> 172.21.15.46:0/1313084540 conn(0x7fe248072230 msgr2=0x7fe2480c6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 shutdown_connections 2023-12-16T21:14:12.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.356+0000 7fe24f2ad640 1 -- 172.21.15.46:0/1313084540 wait complete. 2023-12-16T21:14:12.366 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:14:12.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:12.366 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:12.398 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:12.398 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:14:12.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:12.398 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:14:12.462 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:14:12.462 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:12.482 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:14:12.482 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:12.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.647+0000 7febca8da640 1 Processor -- start 2023-12-16T21:14:12.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.647+0000 7febca8da640 1 -- start start 2023-12-16T21:14:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febca8da640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febc415d330 0x7febc40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febca8da640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febca8da640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40b2b10 con 0x7febc40d3730 2023-12-16T21:14:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febca8da640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc40d3b50 con 0x7febc415d330 2023-12-16T21:14:12.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febca8da640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc40d1590 con 0x7febc40d2910 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febc415d330 0x7febc40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc3fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7febc415d330 0x7febc40b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53226/0 (socket says 172.21.15.46:53226) 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc3fff640 1 -- 172.21.15.46:0/346107459 learned_addr learned my addr 172.21.15.46:0/346107459 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc37fe640 1 --2- 172.21.15.46:0/346107459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc3fff640 1 -- 172.21.15.46:0/346107459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 msgr2=0x7febc40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc3fff640 1 --2- 172.21.15.46:0/346107459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc40d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.648+0000 7febc3fff640 1 -- 172.21.15.46:0/346107459 >> v1:172.21.15.121:6789/0 conn(0x7febc40d3730 legacy=0x7febc40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.649+0000 7febc3fff640 1 -- 172.21.15.46:0/346107459 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40d1870 con 0x7febc415d330 2023-12-16T21:14:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.649+0000 7febc3fff640 1 --2- 172.21.15.46:0/346107459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc40b8b80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7febb4009870 tx=0x7febb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4ee5c2ac650b63 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.649+0000 7febc2ffd640 1 -- 172.21.15.46:0/346107459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4031640 con 0x7febc415d330 2023-12-16T21:14:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.649+0000 7febc2ffd640 1 -- 172.21.15.46:0/346107459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb40317a0 con 0x7febc415d330 2023-12-16T21:14:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.649+0000 7febc2ffd640 1 -- 172.21.15.46:0/346107459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4031b30 con 0x7febc415d330 2023-12-16T21:14:12.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 -- 172.21.15.46:0/346107459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 msgr2=0x7febc40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 --2- 172.21.15.46:0/346107459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc40b8b80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7febb4009870 tx=0x7febb402b8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 -- 172.21.15.46:0/346107459 shutdown_connections 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 --2- 172.21.15.46:0/346107459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc40d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 --2- 172.21.15.46:0/346107459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc40b8b80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 -- 172.21.15.46:0/346107459 >> 172.21.15.46:0/346107459 conn(0x7febc4072020 msgr2=0x7febc4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 -- 172.21.15.46:0/346107459 shutdown_connections 2023-12-16T21:14:12.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.650+0000 7febca8da640 1 -- 172.21.15.46:0/346107459 wait complete. 2023-12-16T21:14:12.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.651+0000 7febca8da640 1 Processor -- start 2023-12-16T21:14:12.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.651+0000 7febca8da640 1 -- start start 2023-12-16T21:14:12.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febca8da640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc4153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febca8da640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febc40d3730 0x7febc41f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febca8da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc41efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febca8da640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc41556a0 con 0x7febc415d330 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febca8da640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc41558d0 con 0x7febc40d3730 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febca8da640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc4155b70 con 0x7febc40d2910 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febc8e77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc41efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febc8e77640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc41efc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53232/0 (socket says 172.21.15.46:53232) 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febc8e77640 1 -- 172.21.15.46:0/630871958 learned_addr learned my addr 172.21.15.46:0/630871958 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febc3fff640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc4153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.652+0000 7febc37fe640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febc40d3730 0x7febc41f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc8e77640 1 -- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 msgr2=0x7febc4153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc8e77640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc4153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc8e77640 1 -- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febc40d3730 msgr2=0x7febc41f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc8e77640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febc40d3730 0x7febc41f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc8e77640 1 -- 172.21.15.46:0/630871958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb40094d0 con 0x7febc415d330 2023-12-16T21:14:12.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc8e77640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc41efc00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7febbc00c6b0 tx=0x7febbc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc003b70 con 0x7febc415d330 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febbc003cf0 con 0x7febc415d330 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc4155e50 con 0x7febc415d330 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc003ea0 con 0x7febc415d330 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.653+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc4156410 con 0x7febc415d330 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.654+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc4156630 con 0x7febc415d330 2023-12-16T21:14:12.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.655+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7febbc005310 con 0x7febc415d330 2023-12-16T21:14:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.655+0000 7febc0ff9640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7febb006ae40 0x7febb006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.655+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febbc0892c0 con 0x7febc415d330 2023-12-16T21:14:12.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.655+0000 7febc3fff640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7febb006ae40 0x7febb006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.656+0000 7febc3fff640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7febb006ae40 0x7febb006d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7febb4036000 tx=0x7febb403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.659+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7febbc0543a0 con 0x7febc415d330 2023-12-16T21:14:12.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.837+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc4069400 con 0x7febc415d330 2023-12-16T21:14:12.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.838+0000 7febc0ff9640 1 -- 172.21.15.46:0/630871958 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7febbc0579c0 con 0x7febc415d330 2023-12-16T21:14:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:12.838 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7febb006ae40 msgr2=0x7febb006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7febb006ae40 0x7febb006d300 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7febb4036000 tx=0x7febb403c040 comp rx=0 tx=0).stop 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 msgr2=0x7febc41efc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc41efc00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7febbc00c6b0 tx=0x7febbc00cb70 comp rx=0 tx=0).stop 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 shutdown_connections 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7febb006ae40 0x7febb006d300 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7febc415d330 0x7febc41efc00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febc40d3730 0x7febc41f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 --2- 172.21.15.46:0/630871958 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febc40d2910 0x7febc4153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 >> 172.21.15.46:0/630871958 conn(0x7febc4072020 msgr2=0x7febc40b4510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 shutdown_connections 2023-12-16T21:14:12.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.840+0000 7febca8da640 1 -- 172.21.15.46:0/630871958 wait complete. 2023-12-16T21:14:12.851 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:12.852 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.961+0000 7f7bc7d68640 1 Processor -- start 2023-12-16T21:14:12.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.962+0000 7f7bc7d68640 1 -- start start 2023-12-16T21:14:12.962 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.962+0000 7f7bc7d68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc015d330 0x7f7bc00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.962+0000 7f7bc7d68640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bc00d2910 0x7f7bc00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.962+0000 7f7bc7d68640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc00b2b10 con 0x7f7bc00d2d90 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.962+0000 7f7bc7d68640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc00d1590 con 0x7f7bc00d2910 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.962+0000 7f7bc7d68640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00d1870 con 0x7f7bc015d330 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc5303640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7bc00d2910 0x7f7bc00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc6305640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7bc00d2d90 0x7f7bc00b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33468/0 (socket says 172.21.15.46:33468) 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc6305640 1 -- 172.21.15.46:0/4020512040 learned_addr learned my addr 172.21.15.46:0/4020512040 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc5b04640 1 --2- 172.21.15.46:0/4020512040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc015d330 0x7f7bc00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc5303640 1 -- 172.21.15.46:0/4020512040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc015d330 msgr2=0x7f7bc00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc5303640 1 --2- 172.21.15.46:0/4020512040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc015d330 0x7f7bc00b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc5303640 1 -- 172.21.15.46:0/4020512040 >> v1:172.21.15.46:6789/0 conn(0x7f7bc00d2d90 legacy=0x7f7bc00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc4b02640 1 -- 172.21.15.46:0/4020512040 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126402602 0 0) 0x7f7bc00b2b10 con 0x7f7bc00d2d90 2023-12-16T21:14:12.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.963+0000 7f7bc5303640 1 -- 172.21.15.46:0/4020512040 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc00d1b50 con 0x7f7bc00d2910 2023-12-16T21:14:12.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc5303640 1 --2- 172.21.15.46:0/4020512040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc00d31f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00097a0 tx=0x7f7bb002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4870574ded277773 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc4b02640 1 -- 172.21.15.46:0/4020512040 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb002e070 con 0x7f7bc00d2910 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc4b02640 1 -- 172.21.15.46:0/4020512040 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb00315a0 con 0x7f7bc00d2910 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc4b02640 1 -- 172.21.15.46:0/4020512040 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb0031930 con 0x7f7bc00d2910 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 -- 172.21.15.46:0/4020512040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 msgr2=0x7f7bc00d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/4020512040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc00d31f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7bb00097a0 tx=0x7f7bb002b840 comp rx=0 tx=0).stop 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 -- 172.21.15.46:0/4020512040 shutdown_connections 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/4020512040 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc00d31f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/4020512040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc015d330 0x7f7bc00b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 -- 172.21.15.46:0/4020512040 >> 172.21.15.46:0/4020512040 conn(0x7f7bc0072020 msgr2=0x7f7bc0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 -- 172.21.15.46:0/4020512040 shutdown_connections 2023-12-16T21:14:12.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.964+0000 7f7bc7d68640 1 -- 172.21.15.46:0/4020512040 wait complete. 2023-12-16T21:14:12.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 Processor -- start 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 -- start start 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc0149c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc00d2d90 0x7f7bc0149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bc015d330 0x7f7bc015f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00cab40 con 0x7f7bc00d2910 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc015f380 con 0x7f7bc015d330 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc7d68640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc00ca330 con 0x7f7bc00d2d90 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc5b04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc0149c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc5b04640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc0149c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53260/0 (socket says 172.21.15.46:53260) 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.966+0000 7f7bc5b04640 1 -- 172.21.15.46:0/1645803228 learned_addr learned my addr 172.21.15.46:0/1645803228 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5303640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc00d2d90 0x7f7bc0149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc6305640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bc015d330 0x7f7bc015f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5b04640 1 -- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc00d2d90 msgr2=0x7f7bc0149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5b04640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc00d2d90 0x7f7bc0149330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5b04640 1 -- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bc015d330 msgr2=0x7f7bc015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5b04640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bc015d330 0x7f7bc015f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5b04640 1 -- 172.21.15.46:0/1645803228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb0009400 con 0x7f7bc00d2910 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc6305640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bc015d330 0x7f7bc015f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc5b04640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc0149c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00d800 tx=0x7f7bbc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00b6c0 con 0x7f7bc00d2910 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bbc00b820 con 0x7f7bc00d2910 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00bbb0 con 0x7f7bc00d2910 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc00cd4e0 con 0x7f7bc00d2910 2023-12-16T21:14:12.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.967+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc00ca860 con 0x7f7bc00d2910 2023-12-16T21:14:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.968+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc0069400 con 0x7f7bc00d2910 2023-12-16T21:14:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.969+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7bbc00bd10 con 0x7f7bc00d2910 2023-12-16T21:14:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.969+0000 7f7bb67fc640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7ba406ae40 0x7f7ba406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:12.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.969+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bbc089b80 con 0x7f7bc00d2910 2023-12-16T21:14:12.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.969+0000 7f7bc5303640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7ba406ae40 0x7f7ba406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:12.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.970+0000 7f7bc5303640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7ba406ae40 0x7f7ba406d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0036000 tx=0x7f7bb003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:12.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:12.973+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7bbc054ba0 con 0x7f7bc00d2910 2023-12-16T21:14:13.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.148+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7bc0159dd0 con 0x7f7bc00d2910 2023-12-16T21:14:13.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.148+0000 7f7bb67fc640 1 -- 172.21.15.46:0/1645803228 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7bbc0581c0 con 0x7f7bc00d2910 2023-12-16T21:14:13.148 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:13.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7ba406ae40 msgr2=0x7f7ba406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7ba406ae40 0x7f7ba406d300 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7bb0036000 tx=0x7f7bb003d040 comp rx=0 tx=0).stop 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 msgr2=0x7f7bc0149c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc0149c80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00d800 tx=0x7f7bbc00dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 shutdown_connections 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7ba406ae40 0x7f7ba406d300 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7bc015d330 0x7f7bc015f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7bc00d2d90 0x7f7bc0149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 --2- 172.21.15.46:0/1645803228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7bc00d2910 0x7f7bc0149c80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 >> 172.21.15.46:0/1645803228 conn(0x7f7bc0072020 msgr2=0x7f7bc00d0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 shutdown_connections 2023-12-16T21:14:13.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.150+0000 7f7bc7d68640 1 -- 172.21.15.46:0/1645803228 wait complete. 2023-12-16T21:14:13.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:13.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.270+0000 7f9e0ceda640 1 Processor -- start 2023-12-16T21:14:13.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.270+0000 7f9e0ceda640 1 -- start start 2023-12-16T21:14:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.271+0000 7f9e0ceda640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9e080d2910 0x7f9e080d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.271+0000 7f9e0ceda640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9e080d3730 0x7f9e080b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.271+0000 7f9e0ceda640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080b2b10 con 0x7f9e0815d330 2023-12-16T21:14:13.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.271+0000 7f9e0ceda640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e080d3bc0 con 0x7f9e080d2910 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.271+0000 7f9e0ceda640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e080d1590 con 0x7f9e080d3730 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e05d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9e080d2910 0x7f9e080d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e0659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9e0815d330 0x7f9e080b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33470/0 (socket says 172.21.15.46:33470) 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e0659c640 1 -- 172.21.15.46:0/4117805489 learned_addr learned my addr 172.21.15.46:0/4117805489 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e06d9d640 1 --2- 172.21.15.46:0/4117805489 >> v2:172.21.15.121:3300/0 conn(0x7f9e080d3730 0x7f9e080b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e05d9b640 1 -- 172.21.15.46:0/4117805489 >> v2:172.21.15.121:3300/0 conn(0x7f9e080d3730 msgr2=0x7f9e080b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e05d9b640 1 --2- 172.21.15.46:0/4117805489 >> v2:172.21.15.121:3300/0 conn(0x7f9e080d3730 0x7f9e080b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e05d9b640 1 -- 172.21.15.46:0/4117805489 >> v1:172.21.15.46:6789/0 conn(0x7f9e0815d330 legacy=0x7f9e080b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e05d9b640 1 -- 172.21.15.46:0/4117805489 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080d1870 con 0x7f9e080d2910 2023-12-16T21:14:13.272 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.272+0000 7f9e0559a640 1 -- 172.21.15.46:0/4117805489 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505154650 0 0) 0x7f9e080b2b10 con 0x7f9e0815d330 2023-12-16T21:14:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.273+0000 7f9e05d9b640 1 --2- 172.21.15.46:0/4117805489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e080d31f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9e0806c500 tx=0x7f9dfc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f76dc58eb28224 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.273+0000 7f9e0559a640 1 -- 172.21.15.46:0/4117805489 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc02e070 con 0x7f9e080d2910 2023-12-16T21:14:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.273+0000 7f9e0559a640 1 -- 172.21.15.46:0/4117805489 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0315a0 con 0x7f9e080d2910 2023-12-16T21:14:13.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.273+0000 7f9e0559a640 1 -- 172.21.15.46:0/4117805489 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc031930 con 0x7f9e080d2910 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.273+0000 7f9e0ceda640 1 -- 172.21.15.46:0/4117805489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 msgr2=0x7f9e080d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.273+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/4117805489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e080d31f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9e0806c500 tx=0x7f9dfc02b840 comp rx=0 tx=0).stop 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.274+0000 7f9e0ceda640 1 -- 172.21.15.46:0/4117805489 shutdown_connections 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.274+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/4117805489 >> v2:172.21.15.121:3300/0 conn(0x7f9e080d3730 0x7f9e080b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.274+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/4117805489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e080d31f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.274+0000 7f9e0ceda640 1 -- 172.21.15.46:0/4117805489 >> 172.21.15.46:0/4117805489 conn(0x7f9e08072020 msgr2=0x7f9e08072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.274+0000 7f9e0ceda640 1 -- 172.21.15.46:0/4117805489 shutdown_connections 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.274+0000 7f9e0ceda640 1 -- 172.21.15.46:0/4117805489 wait complete. 2023-12-16T21:14:13.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.275+0000 7f9e0ceda640 1 Processor -- start 2023-12-16T21:14:13.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.275+0000 7f9e0ceda640 1 -- start start 2023-12-16T21:14:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e081532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e080d3730 0x7f9e0815d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0815d330 0x7f9e08157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08158990 con 0x7f9e080d2910 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08158c70 con 0x7f9e0815d330 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08158f50 con 0x7f9e080d3730 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e081532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e081532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53274/0 (socket says 172.21.15.46:53274) 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 -- 172.21.15.46:0/2068207354 learned_addr learned my addr 172.21.15.46:0/2068207354 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e05d9b640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e080d3730 0x7f9e0815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e06d9d640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0815d330 0x7f9e08157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 -- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e080d3730 msgr2=0x7f9e0815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e080d3730 0x7f9e0815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 -- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0815d330 msgr2=0x7f9e08157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0815d330 0x7f9e08157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 -- 172.21.15.46:0/2068207354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc009400 con 0x7f9e080d2910 2023-12-16T21:14:13.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0659c640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e081532b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c430 tx=0x7f9df800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df80074e0 con 0x7f9e080d2910 2023-12-16T21:14:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df8007660 con 0x7f9e080d2910 2023-12-16T21:14:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08159230 con 0x7f9e080d2910 2023-12-16T21:14:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.276+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df8004a20 con 0x7f9e080d2910 2023-12-16T21:14:13.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.277+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0815dc50 con 0x7f9e080d2910 2023-12-16T21:14:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.277+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e08069400 con 0x7f9e080d2910 2023-12-16T21:14:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.278+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9df8004b80 con 0x7f9e080d2910 2023-12-16T21:14:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.278+0000 7f9df6ffd640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9de806ae40 0x7f9de806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.278+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9df8089270 con 0x7f9e080d2910 2023-12-16T21:14:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.278+0000 7f9e05d9b640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9de806ae40 0x7f9de806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.279+0000 7f9e05d9b640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9de806ae40 0x7f9de806d300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc036000 tx=0x7f9dfc03e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.283+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9df8054460 con 0x7f9e080d2910 2023-12-16T21:14:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.457+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9e0815de40 con 0x7f9e080d2910 2023-12-16T21:14:13.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.457+0000 7f9df6ffd640 1 -- 172.21.15.46:0/2068207354 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9df8057a80 con 0x7f9e080d2910 2023-12-16T21:14:13.457 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9de806ae40 msgr2=0x7f9de806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9de806ae40 0x7f9de806d300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc036000 tx=0x7f9dfc03e040 comp rx=0 tx=0).stop 2023-12-16T21:14:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 msgr2=0x7f9e081532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e081532b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c430 tx=0x7f9df800c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 shutdown_connections 2023-12-16T21:14:13.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9de806ae40 0x7f9de806d300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e0815d330 0x7f9e08157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e080d3730 0x7f9e0815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 --2- 172.21.15.46:0/2068207354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9e080d2910 0x7f9e081532b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 >> 172.21.15.46:0/2068207354 conn(0x7f9e08072020 msgr2=0x7f9e080b6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 shutdown_connections 2023-12-16T21:14:13.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.459+0000 7f9e0ceda640 1 -- 172.21.15.46:0/2068207354 wait complete. 2023-12-16T21:14:13.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:13.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.580+0000 7fd0dda61640 1 Processor -- start 2023-12-16T21:14:13.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.580+0000 7fd0dda61640 1 -- start start 2023-12-16T21:14:13.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0dda61640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd0d8150560 0x7fd0d80b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0dda61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d80b3390 0x7fd0d815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0dda61640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0d8150940 0x7fd0d815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0dda61640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d80b49f0 con 0x7fd0d8150940 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0dda61640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d80b3de0 con 0x7fd0d8150560 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0dda61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d80b40c0 con 0x7fd0d80b3390 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0d8150940 0x7fd0d815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd0d8150940 0x7fd0d815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53290/0 (socket says 172.21.15.46:53290) 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 -- 172.21.15.46:0/3152586012 learned_addr learned my addr 172.21.15.46:0/3152586012 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d80b3390 0x7fd0d815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d6ffd640 1 --2- 172.21.15.46:0/3152586012 >> v2:172.21.15.121:3300/0 conn(0x7fd0d8150560 0x7fd0d80b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 -- 172.21.15.46:0/3152586012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d80b3390 msgr2=0x7fd0d815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d80b3390 0x7fd0d815d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 -- 172.21.15.46:0/3152586012 >> v2:172.21.15.121:3300/0 conn(0x7fd0d8150560 msgr2=0x7fd0d80b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 --2- 172.21.15.46:0/3152586012 >> v2:172.21.15.121:3300/0 conn(0x7fd0d8150560 0x7fd0d80b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0d77fe640 1 -- 172.21.15.46:0/3152586012 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d80b43a0 con 0x7fd0d8150940 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.581+0000 7fd0cffff640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d80b3390 0x7fd0d815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:13.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.582+0000 7fd0d77fe640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d8150940 0x7fd0d815d740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8072840 tx=0x7fd0d002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de5b759313b3349 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.582+0000 7fd0d67fc640 1 -- 172.21.15.46:0/3152586012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d002ee90 con 0x7fd0d8150940 2023-12-16T21:14:13.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.582+0000 7fd0d67fc640 1 -- 172.21.15.46:0/3152586012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d0032070 con 0x7fd0d8150940 2023-12-16T21:14:13.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.582+0000 7fd0d67fc640 1 -- 172.21.15.46:0/3152586012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0038620 con 0x7fd0d8150940 2023-12-16T21:14:13.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 -- 172.21.15.46:0/3152586012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d8150940 msgr2=0x7fd0d815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d8150940 0x7fd0d815d740 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8072840 tx=0x7fd0d002e0e0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 -- 172.21.15.46:0/3152586012 shutdown_connections 2023-12-16T21:14:13.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d8150940 0x7fd0d815d740 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 --2- 172.21.15.46:0/3152586012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d80b3390 0x7fd0d815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 --2- 172.21.15.46:0/3152586012 >> v2:172.21.15.121:3300/0 conn(0x7fd0d8150560 0x7fd0d80b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 -- 172.21.15.46:0/3152586012 >> 172.21.15.46:0/3152586012 conn(0x7fd0d806a4d0 msgr2=0x7fd0d806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 -- 172.21.15.46:0/3152586012 shutdown_connections 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.583+0000 7fd0dda61640 1 -- 172.21.15.46:0/3152586012 wait complete. 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.584+0000 7fd0dda61640 1 Processor -- start 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.584+0000 7fd0dda61640 1 -- start start 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.584+0000 7fd0dda61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 0x7fd0d8153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.584+0000 7fd0dda61640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0d8150560 0x7fd0d8148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0dda61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d8148e90 0x7fd0d8158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0dda61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8149340 con 0x7fd0d80b3390 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0dda61640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8154fc0 con 0x7fd0d8150560 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0dda61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d8159920 con 0x7fd0d8148e90 2023-12-16T21:14:13.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 0x7fd0d8153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 0x7fd0d8153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53304/0 (socket says 172.21.15.46:53304) 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 -- 172.21.15.46:0/71202708 learned_addr learned my addr 172.21.15.46:0/71202708 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d77fe640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d8148e90 0x7fd0d8158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0cffff640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0d8150560 0x7fd0d8148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 -- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d8148e90 msgr2=0x7fd0d8158bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d8148e90 0x7fd0d8158bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 -- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0d8150560 msgr2=0x7fd0d8148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0d8150560 0x7fd0d8148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 -- 172.21.15.46:0/71202708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d000bd40 con 0x7fd0d80b3390 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d6ffd640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 0x7fd0d8153340 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800e9b0 tx=0x7fd0c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8009670 con 0x7fd0d80b3390 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c800c9a0 con 0x7fd0d80b3390 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d8159c00 con 0x7fd0d80b3390 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800cd50 con 0x7fd0d80b3390 2023-12-16T21:14:13.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.585+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d8150d30 con 0x7fd0d80b3390 2023-12-16T21:14:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.587+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d8069400 con 0x7fd0d80b3390 2023-12-16T21:14:13.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.587+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd0c8013470 con 0x7fd0d80b3390 2023-12-16T21:14:13.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.587+0000 7fd0d4ff9640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd0b806ae40 0x7fd0b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.587+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0c8089710 con 0x7fd0d80b3390 2023-12-16T21:14:13.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.587+0000 7fd0cffff640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd0b806ae40 0x7fd0b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.588+0000 7fd0cffff640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd0b806ae40 0x7fd0b806d300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd0d806ab20 tx=0x7fd0c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.592+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd0c8054870 con 0x7fd0d80b3390 2023-12-16T21:14:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.770+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd0d814ffa0 con 0x7fd0d80b3390 2023-12-16T21:14:13.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.770+0000 7fd0d4ff9640 1 -- 172.21.15.46:0/71202708 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd0c8057e90 con 0x7fd0d80b3390 2023-12-16T21:14:13.770 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd0b806ae40 msgr2=0x7fd0b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd0b806ae40 0x7fd0b806d300 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd0d806ab20 tx=0x7fd0c0008040 comp rx=0 tx=0).stop 2023-12-16T21:14:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 msgr2=0x7fd0d8153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 0x7fd0d8153340 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800e9b0 tx=0x7fd0c800ee70 comp rx=0 tx=0).stop 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 shutdown_connections 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd0b806ae40 0x7fd0b806d300 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0d8148e90 0x7fd0d8158bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0d8150560 0x7fd0d8148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 --2- 172.21.15.46:0/71202708 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0d80b3390 0x7fd0d8153340 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.772+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 >> 172.21.15.46:0/71202708 conn(0x7fd0d806a4d0 msgr2=0x7fd0d80b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.773+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 shutdown_connections 2023-12-16T21:14:13.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.773+0000 7fd0dda61640 1 -- 172.21.15.46:0/71202708 wait complete. 2023-12-16T21:14:13.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:13.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.891+0000 7fd25d9bb640 1 Processor -- start 2023-12-16T21:14:13.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.891+0000 7fd25d9bb640 1 -- start start 2023-12-16T21:14:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd25d9bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd25815b130 0x7fd2580d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd25d9bb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2580cf320 0x7fd2580cd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd25d9bb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2580b3f10 con 0x7fd25815f760 2023-12-16T21:14:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd25d9bb640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25815fd20 con 0x7fd2580cf320 2023-12-16T21:14:13.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd25d9bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2580d0730 con 0x7fd25815b130 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd2577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2580cf320 0x7fd2580cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd2577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2580cf320 0x7fd2580cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53314/0 (socket says 172.21.15.46:53314) 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd2577fe640 1 -- 172.21.15.46:0/3405988951 learned_addr learned my addr 172.21.15.46:0/3405988951 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd256ffd640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd25815b130 0x7fd2580d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.892+0000 7fd2577fe640 1 -- 172.21.15.46:0/3405988951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd25815b130 msgr2=0x7fd2580d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd2577fe640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd25815b130 0x7fd2580d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd2577fe640 1 -- 172.21.15.46:0/3405988951 >> v1:172.21.15.121:6789/0 conn(0x7fd25815f760 legacy=0x7fd2580cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd2577fe640 1 -- 172.21.15.46:0/3405988951 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2580d0a50 con 0x7fd2580cf320 2023-12-16T21:14:13.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd256ffd640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd25815b130 0x7fd2580d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:13.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd2577fe640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2580cf320 0x7fd2580cd310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd248009e80 tx=0x7fd24802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b45b50dcdcdc38 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd255ffb640 1 -- 172.21.15.46:0/3405988951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24802ca50 con 0x7fd2580cf320 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.893+0000 7fd255ffb640 1 -- 172.21.15.46:0/3405988951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd24802cbb0 con 0x7fd2580cf320 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd255ffb640 1 -- 172.21.15.46:0/3405988951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2480343f0 con 0x7fd2580cf320 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 -- 172.21.15.46:0/3405988951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2580cf320 msgr2=0x7fd2580cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2580cf320 0x7fd2580cd310 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd248009e80 tx=0x7fd24802beb0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 -- 172.21.15.46:0/3405988951 shutdown_connections 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2580cf320 0x7fd2580cd310 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/3405988951 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd25815b130 0x7fd2580d01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 -- 172.21.15.46:0/3405988951 >> 172.21.15.46:0/3405988951 conn(0x7fd2580b2c10 msgr2=0x7fd2580b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 -- 172.21.15.46:0/3405988951 shutdown_connections 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.894+0000 7fd25d9bb640 1 -- 172.21.15.46:0/3405988951 wait complete. 2023-12-16T21:14:13.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.895+0000 7fd25d9bb640 1 Processor -- start 2023-12-16T21:14:13.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.895+0000 7fd25d9bb640 1 -- start start 2023-12-16T21:14:13.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd25d9bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2580cf320 0x7fd2581f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd25d9bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 0x7fd2580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd25d9bb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd25815f760 0x7fd2580cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd25d9bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2581f0a50 con 0x7fd25815b130 2023-12-16T21:14:13.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd25d9bb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd258150d80 con 0x7fd25815f760 2023-12-16T21:14:13.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd25d9bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2581f6600 con 0x7fd2580cf320 2023-12-16T21:14:13.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd2567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 0x7fd2580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd2567fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 0x7fd2580ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53320/0 (socket says 172.21.15.46:53320) 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd2567fc640 1 -- 172.21.15.46:0/2436542967 learned_addr learned my addr 172.21.15.46:0/2436542967 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd256ffd640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2580cf320 0x7fd2581f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.896+0000 7fd2577fe640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd25815f760 0x7fd2580cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd2567fc640 1 -- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2580cf320 msgr2=0x7fd2581f0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd2567fc640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2580cf320 0x7fd2581f0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd2567fc640 1 -- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd25815f760 msgr2=0x7fd2580cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd2567fc640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd25815f760 0x7fd2580cec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:13.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd2567fc640 1 -- 172.21.15.46:0/2436542967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd248009b30 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd2567fc640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 0x7fd2580ce6c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd24000b3f0 tx=0x7fd24000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011830 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240011990 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2581f68e0 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011d40 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.897+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2581f6ea0 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.898+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd258067fc0 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.898+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd24001a400 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.899+0000 7fd24ffff640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd23806ae70 0x7fd23806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.899+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd240089340 con 0x7fd25815b130 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.899+0000 7fd256ffd640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd23806ae70 0x7fd23806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:13.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.900+0000 7fd256ffd640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd23806ae70 0x7fd23806d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd25000ab60 tx=0x7fd250008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:13.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:13.903+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd240054490 con 0x7fd25815b130 2023-12-16T21:14:14.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.093+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd258067420 con 0x7fd25815b130 2023-12-16T21:14:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.093+0000 7fd24ffff640 1 -- 172.21.15.46:0/2436542967 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd240057ab0 con 0x7fd25815b130 2023-12-16T21:14:14.093 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd23806ae70 msgr2=0x7fd23806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd23806ae70 0x7fd23806d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd25000ab60 tx=0x7fd250008040 comp rx=0 tx=0).stop 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 msgr2=0x7fd2580ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 0x7fd2580ce6c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd24000b3f0 tx=0x7fd24000b8b0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 shutdown_connections 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd23806ae70 0x7fd23806d330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd25815b130 0x7fd2580ce6c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd25815f760 0x7fd2580cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 --2- 172.21.15.46:0/2436542967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2580cf320 0x7fd2581f0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 >> 172.21.15.46:0/2436542967 conn(0x7fd2580b2c10 msgr2=0x7fd2580cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 shutdown_connections 2023-12-16T21:14:14.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.095+0000 7fd25d9bb640 1 -- 172.21.15.46:0/2436542967 wait complete. 2023-12-16T21:14:14.106 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:14:14.106 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:14.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.232+0000 7f7d52db0640 1 Processor -- start 2023-12-16T21:14:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.232+0000 7f7d52db0640 1 -- start start 2023-12-16T21:14:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d52db0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d4c160fb0 0x7f7d4c161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d52db0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d4c0d0ed0 0x7f7d4c0d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d52db0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c161d20 con 0x7f7d4c06ea50 2023-12-16T21:14:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d52db0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c15e760 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d52db0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c15d9a0 con 0x7f7d4c160fb0 2023-12-16T21:14:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d5134d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d4c0d0ed0 0x7f7d4c0d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d50b4c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7d4c06ea50 0x7f7d4c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33480/0 (socket says 172.21.15.46:33480) 2023-12-16T21:14:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d50b4c640 1 -- 172.21.15.46:0/3756321753 learned_addr learned my addr 172.21.15.46:0/3756321753 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d4bfff640 1 --2- 172.21.15.46:0/3756321753 >> v2:172.21.15.121:3300/0 conn(0x7f7d4c160fb0 0x7f7d4c161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d5134d640 1 -- 172.21.15.46:0/3756321753 >> v2:172.21.15.121:3300/0 conn(0x7f7d4c160fb0 msgr2=0x7f7d4c161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d5134d640 1 --2- 172.21.15.46:0/3756321753 >> v2:172.21.15.121:3300/0 conn(0x7f7d4c160fb0 0x7f7d4c161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d5134d640 1 -- 172.21.15.46:0/3756321753 >> v1:172.21.15.46:6789/0 conn(0x7f7d4c06ea50 legacy=0x7f7d4c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.233+0000 7f7d5134d640 1 -- 172.21.15.46:0/3756321753 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c15dc80 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.234+0000 7f7d5134d640 1 --2- 172.21.15.46:0/3756321753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c0d1320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0b6690 tx=0x7f7d3402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ac8fbd0f73046f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.234+0000 7f7d4b7fe640 1 -- 172.21.15.46:0/3756321753 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3402e070 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.234+0000 7f7d4b7fe640 1 -- 172.21.15.46:0/3756321753 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d340315a0 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.234+0000 7f7d4b7fe640 1 -- 172.21.15.46:0/3756321753 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34031930 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 -- 172.21.15.46:0/3756321753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 msgr2=0x7f7d4c0d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 --2- 172.21.15.46:0/3756321753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c0d1320 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0b6690 tx=0x7f7d3402b840 comp rx=0 tx=0).stop 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 -- 172.21.15.46:0/3756321753 shutdown_connections 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 --2- 172.21.15.46:0/3756321753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c0d1320 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 --2- 172.21.15.46:0/3756321753 >> v2:172.21.15.121:3300/0 conn(0x7f7d4c160fb0 0x7f7d4c161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 -- 172.21.15.46:0/3756321753 >> 172.21.15.46:0/3756321753 conn(0x7f7d4c06a6b0 msgr2=0x7f7d4c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:14.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 -- 172.21.15.46:0/3756321753 shutdown_connections 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.235+0000 7f7d52db0640 1 -- 172.21.15.46:0/3756321753 wait complete. 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.236+0000 7f7d52db0640 1 Processor -- start 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.236+0000 7f7d52db0640 1 -- start start 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d52db0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4c06ea50 0x7f7d4c1f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d52db0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d52db0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4c160fb0 0x7f7d4c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d52db0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c1f3250 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d52db0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c13e500 con 0x7f7d4c06ea50 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d52db0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4c13e220 con 0x7f7d4c160fb0 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53342/0 (socket says 172.21.15.46:53342) 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 -- 172.21.15.46:0/4221401060 learned_addr learned my addr 172.21.15.46:0/4221401060 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 -- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4c160fb0 msgr2=0x7f7d4c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4c160fb0 0x7f7d4c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 -- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4c06ea50 msgr2=0x7f7d4c1f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4c06ea50 0x7f7d4c1f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.237+0000 7f7d4bfff640 1 -- 172.21.15.46:0/4221401060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34009400 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.238+0000 7f7d4bfff640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c156b90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00d800 tx=0x7f7d3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.238+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c00b6c0 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.238+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3c00b820 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.238+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c00bbb0 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.238+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c0b2e70 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.238+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c0b3430 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.239+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c0b35f0 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.239+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7d3c00bd10 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.240+0000 7f7d497fa640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d2c06ae40 0x7f7d2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.240+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d3c089bc0 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.240+0000 7f7d50b4c640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d2c06ae40 0x7f7d2c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.241+0000 7f7d50b4c640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d2c06ae40 0x7f7d2c06d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0d0c70 tx=0x7f7d440015c0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.244+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7d3c054c70 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.421+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d4c15b290 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.422+0000 7f7d497fa640 1 -- 172.21.15.46:0/4221401060 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f7d3c058290 con 0x7f7d4c0d0ed0 2023-12-16T21:14:14.461 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:14.461 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d2c06ae40 msgr2=0x7f7d2c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d2c06ae40 0x7f7d2c06d300 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0d0c70 tx=0x7f7d440015c0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 msgr2=0x7f7d4c156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c156b90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c00d800 tx=0x7f7d3c00dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 shutdown_connections 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d2c06ae40 0x7f7d2c06d300 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d4c0d0ed0 0x7f7d4c156b90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d4c160fb0 0x7f7d4c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 --2- 172.21.15.46:0/4221401060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d4c06ea50 0x7f7d4c1f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 >> 172.21.15.46:0/4221401060 conn(0x7f7d4c06a6b0 msgr2=0x7f7d4c0b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 shutdown_connections 2023-12-16T21:14:14.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.424+0000 7f7d52db0640 1 -- 172.21.15.46:0/4221401060 wait complete. 2023-12-16T21:14:14.464 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:14.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:14.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.574+0000 7f5c068c9640 1 Processor -- start 2023-12-16T21:14:14.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.574+0000 7f5c068c9640 1 -- start start 2023-12-16T21:14:14.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5c068c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c0015d270 0x7f5c0006ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5c068c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5c000cdee0 0x7f5c000cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5c068c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c000d0c40 con 0x7f5c000709c0 2023-12-16T21:14:14.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5c068c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c000ced90 con 0x7f5c0015d270 2023-12-16T21:14:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5c068c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c000cf070 con 0x7f5c000cdee0 2023-12-16T21:14:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5bfffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5c0015d270 0x7f5c0006ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5bff7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5c000709c0 0x7f5c000cd7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33496/0 (socket says 172.21.15.46:33496) 2023-12-16T21:14:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.575+0000 7f5bff7fe640 1 -- 172.21.15.46:0/3416431310 learned_addr learned my addr 172.21.15.46:0/3416431310 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5c04e66640 1 --2- 172.21.15.46:0/3416431310 >> v2:172.21.15.121:3300/0 conn(0x7f5c000cdee0 0x7f5c000cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5bfffff640 1 -- 172.21.15.46:0/3416431310 >> v2:172.21.15.121:3300/0 conn(0x7f5c000cdee0 msgr2=0x7f5c000cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5bfffff640 1 --2- 172.21.15.46:0/3416431310 >> v2:172.21.15.121:3300/0 conn(0x7f5c000cdee0 0x7f5c000cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5bfffff640 1 -- 172.21.15.46:0/3416431310 >> v1:172.21.15.46:6789/0 conn(0x7f5c000709c0 legacy=0x7f5c000cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5bfeffd640 1 -- 172.21.15.46:0/3416431310 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708147029 0 0) 0x7f5c000d0c40 con 0x7f5c000709c0 2023-12-16T21:14:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5bfffff640 1 -- 172.21.15.46:0/3416431310 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c000d08f0 con 0x7f5c0015d270 2023-12-16T21:14:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.576+0000 7f5bfffff640 1 --2- 172.21.15.46:0/3416431310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c0015d270 0x7f5c0006ce70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5c000ceba0 tx=0x7f5bf002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d5ed9e5e6fd6e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5bfeffd640 1 -- 172.21.15.46:0/3416431310 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf002e070 con 0x7f5c0015d270 2023-12-16T21:14:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5bfeffd640 1 -- 172.21.15.46:0/3416431310 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf00315a0 con 0x7f5c0015d270 2023-12-16T21:14:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5bfeffd640 1 -- 172.21.15.46:0/3416431310 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0031930 con 0x7f5c0015d270 2023-12-16T21:14:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 -- 172.21.15.46:0/3416431310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c0015d270 msgr2=0x7f5c0006ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 --2- 172.21.15.46:0/3416431310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c0015d270 0x7f5c0006ce70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5c000ceba0 tx=0x7f5bf002b840 comp rx=0 tx=0).stop 2023-12-16T21:14:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 -- 172.21.15.46:0/3416431310 shutdown_connections 2023-12-16T21:14:14.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 --2- 172.21.15.46:0/3416431310 >> v2:172.21.15.121:3300/0 conn(0x7f5c000cdee0 0x7f5c000cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 --2- 172.21.15.46:0/3416431310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c0015d270 0x7f5c0006ce70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 -- 172.21.15.46:0/3416431310 >> 172.21.15.46:0/3416431310 conn(0x7f5c000b2c10 msgr2=0x7f5c000b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 -- 172.21.15.46:0/3416431310 shutdown_connections 2023-12-16T21:14:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.577+0000 7f5c068c9640 1 -- 172.21.15.46:0/3416431310 wait complete. 2023-12-16T21:14:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.578+0000 7f5c068c9640 1 Processor -- start 2023-12-16T21:14:14.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 -- start start 2023-12-16T21:14:14.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c000709c0 0x7f5c00148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 0x7f5c000c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c0015d270 0x7f5c001f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c000ca5c0 con 0x7f5c000cdee0 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c000ca8a0 con 0x7f5c000709c0 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5c068c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c000cab80 con 0x7f5c0015d270 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5bff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 0x7f5c000c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5bff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 0x7f5c000c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53356/0 (socket says 172.21.15.46:53356) 2023-12-16T21:14:14.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5bff7fe640 1 -- 172.21.15.46:0/2660381320 learned_addr learned my addr 172.21.15.46:0/2660381320 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.579+0000 7f5bfffff640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c000709c0 0x7f5c00148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5c04e66640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c0015d270 0x7f5c001f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bff7fe640 1 -- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c0015d270 msgr2=0x7f5c001f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bff7fe640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c0015d270 0x7f5c001f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bff7fe640 1 -- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c000709c0 msgr2=0x7f5c00148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bff7fe640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c000709c0 0x7f5c00148870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bff7fe640 1 -- 172.21.15.46:0/2660381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bf0009400 con 0x7f5c000cdee0 2023-12-16T21:14:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5c04e66640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c0015d270 0x7f5c001f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bff7fe640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 0x7f5c000c9b70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c4d0 tx=0x7f5bf400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40074e0 con 0x7f5c000cdee0 2023-12-16T21:14:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf4007660 con 0x7f5c000cdee0 2023-12-16T21:14:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4004a20 con 0x7f5c000cdee0 2023-12-16T21:14:14.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c000c61b0 con 0x7f5c000cdee0 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.580+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c000c5e60 con 0x7f5c000cdee0 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.581+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c00067fc0 con 0x7f5c000cdee0 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.581+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5bf4004b80 con 0x7f5c000cdee0 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.582+0000 7f5bfcff9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5bec06ae40 0x7f5bec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.582+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bf4088710 con 0x7f5c000cdee0 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.582+0000 7f5bfffff640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5bec06ae40 0x7f5bec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.583+0000 7f5bfffff640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5bec06ae40 0x7f5bec06d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0036000 tx=0x7f5bf003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.586+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5bf4053870 con 0x7f5c000cdee0 2023-12-16T21:14:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.761+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c000c63a0 con 0x7f5c000cdee0 2023-12-16T21:14:14.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.762+0000 7f5bfcff9640 1 -- 172.21.15.46:0/2660381320 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5bf40180a0 con 0x7f5c000cdee0 2023-12-16T21:14:14.762 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5bec06ae40 msgr2=0x7f5bec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5bec06ae40 0x7f5bec06d300 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0036000 tx=0x7f5bf003d040 comp rx=0 tx=0).stop 2023-12-16T21:14:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 msgr2=0x7f5c000c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 0x7f5c000c9b70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400c4d0 tx=0x7f5bf400c990 comp rx=0 tx=0).stop 2023-12-16T21:14:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 shutdown_connections 2023-12-16T21:14:14.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5bec06ae40 0x7f5bec06d300 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c000cdee0 0x7f5c000c9b70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c0015d270 0x7f5c001f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 --2- 172.21.15.46:0/2660381320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c000709c0 0x7f5c00148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 >> 172.21.15.46:0/2660381320 conn(0x7f5c000b2c10 msgr2=0x7f5c000733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 shutdown_connections 2023-12-16T21:14:14.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.764+0000 7f5c068c9640 1 -- 172.21.15.46:0/2660381320 wait complete. 2023-12-16T21:14:14.774 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:14.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.884+0000 7f7d9823d640 1 Processor -- start 2023-12-16T21:14:14.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d9823d640 1 -- start start 2023-12-16T21:14:14.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d9823d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d900d2910 0x7f7d900d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d9823d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d900d3730 0x7f7d900b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d9823d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d900b2b10 con 0x7f7d9015d330 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d9823d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d900d3bc0 con 0x7f7d900d2910 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d9823d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d900d1590 con 0x7f7d900d3730 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d957d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d900d2910 0x7f7d900d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.885+0000 7f7d957d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7d900d2910 0x7f7d900d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53360/0 (socket says 172.21.15.46:53360) 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d957d8640 1 -- 172.21.15.46:0/1896795250 learned_addr learned my addr 172.21.15.46:0/1896795250 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d967da640 1 --2- 172.21.15.46:0/1896795250 >> v2:172.21.15.121:3300/0 conn(0x7f7d900d3730 0x7f7d900b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d957d8640 1 -- 172.21.15.46:0/1896795250 >> v2:172.21.15.121:3300/0 conn(0x7f7d900d3730 msgr2=0x7f7d900b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d957d8640 1 --2- 172.21.15.46:0/1896795250 >> v2:172.21.15.121:3300/0 conn(0x7f7d900d3730 0x7f7d900b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d957d8640 1 -- 172.21.15.46:0/1896795250 >> v1:172.21.15.121:6789/0 conn(0x7f7d9015d330 legacy=0x7f7d900b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d957d8640 1 -- 172.21.15.46:0/1896795250 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d900d1870 con 0x7f7d900d2910 2023-12-16T21:14:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.886+0000 7f7d967da640 1 --2- 172.21.15.46:0/1896795250 >> v2:172.21.15.121:3300/0 conn(0x7f7d900d3730 0x7f7d900b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d957d8640 1 --2- 172.21.15.46:0/1896795250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900d31f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7d9006c500 tx=0x7f7d8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a90c25dc18bf01 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d94fd7640 1 -- 172.21.15.46:0/1896795250 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80031870 con 0x7f7d900d2910 2023-12-16T21:14:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d94fd7640 1 -- 172.21.15.46:0/1896795250 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d800319d0 con 0x7f7d900d2910 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d94fd7640 1 -- 172.21.15.46:0/1896795250 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80031d60 con 0x7f7d900d2910 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 -- 172.21.15.46:0/1896795250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 msgr2=0x7f7d900d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1896795250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900d31f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7d9006c500 tx=0x7f7d8002b910 comp rx=0 tx=0).stop 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 -- 172.21.15.46:0/1896795250 shutdown_connections 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1896795250 >> v2:172.21.15.121:3300/0 conn(0x7f7d900d3730 0x7f7d900b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1896795250 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900d31f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 -- 172.21.15.46:0/1896795250 >> 172.21.15.46:0/1896795250 conn(0x7f7d90072020 msgr2=0x7f7d90072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 -- 172.21.15.46:0/1896795250 shutdown_connections 2023-12-16T21:14:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.887+0000 7f7d9823d640 1 -- 172.21.15.46:0/1896795250 wait complete. 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 Processor -- start 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 -- start start 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d900d3730 0x7f7d9014b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9015d330 0x7f7d900c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900c7470 con 0x7f7d900d2910 2023-12-16T21:14:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9014bc50 con 0x7f7d9015d330 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d9823d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900c6ba0 con 0x7f7d900d3730 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d95fd9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d95fd9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53362/0 (socket says 172.21.15.46:53362) 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.889+0000 7f7d95fd9640 1 -- 172.21.15.46:0/1761545511 learned_addr learned my addr 172.21.15.46:0/1761545511 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d967da640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9015d330 0x7f7d900c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d957d8640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d900d3730 0x7f7d9014b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d95fd9640 1 -- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d900d3730 msgr2=0x7f7d9014b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d95fd9640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d900d3730 0x7f7d9014b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d95fd9640 1 -- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9015d330 msgr2=0x7f7d900c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d95fd9640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9015d330 0x7f7d900c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d95fd9640 1 -- 172.21.15.46:0/1761545511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d800094d0 con 0x7f7d900d2910 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d967da640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9015d330 0x7f7d900c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d95fd9640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900cad40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c400 tx=0x7f7d8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c0052d0 con 0x7f7d900d2910 2023-12-16T21:14:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c005430 con 0x7f7d900d2910 2023-12-16T21:14:14.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d900c9a40 con 0x7f7d900d2910 2023-12-16T21:14:14.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c003830 con 0x7f7d900d2910 2023-12-16T21:14:14.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.890+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d900c6db0 con 0x7f7d900d2910 2023-12-16T21:14:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.891+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d900c75b0 con 0x7f7d900d2910 2023-12-16T21:14:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.892+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7d8c003990 con 0x7f7d900d2910 2023-12-16T21:14:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.892+0000 7f7d867fc640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d7406ae40 0x7f7d7406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.892+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d8c088970 con 0x7f7d900d2910 2023-12-16T21:14:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.892+0000 7f7d957d8640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d7406ae40 0x7f7d7406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.893+0000 7f7d957d8640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d7406ae40 0x7f7d7406d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7d80036000 tx=0x7f7d8003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:14.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:14.896+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7d8c053ab0 con 0x7f7d900d2910 2023-12-16T21:14:15.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.073+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d900c9a40 con 0x7f7d900d2910 2023-12-16T21:14:15.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.073+0000 7f7d867fc640 1 -- 172.21.15.46:0/1761545511 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d8c017020 con 0x7f7d900d2910 2023-12-16T21:14:15.073 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d7406ae40 msgr2=0x7f7d7406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d7406ae40 0x7f7d7406d300 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7d80036000 tx=0x7f7d8003c040 comp rx=0 tx=0).stop 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 msgr2=0x7f7d900cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900cad40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c400 tx=0x7f7d8c00c8c0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 shutdown_connections 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7d7406ae40 0x7f7d7406d300 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d9015d330 0x7f7d900c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d900d3730 0x7f7d9014b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 --2- 172.21.15.46:0/1761545511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d900d2910 0x7f7d900cad40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 >> 172.21.15.46:0/1761545511 conn(0x7f7d90072020 msgr2=0x7f7d90072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 shutdown_connections 2023-12-16T21:14:15.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.075+0000 7f7d9823d640 1 -- 172.21.15.46:0/1761545511 wait complete. 2023-12-16T21:14:15.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:15.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.195+0000 7fce041c9640 1 Processor -- start 2023-12-16T21:14:15.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.195+0000 7fce041c9640 1 -- start start 2023-12-16T21:14:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce041c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcdfc0ceb70 0x7fcdfc0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce041c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcdfc0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce041c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdfc071770 con 0x7fcdfc144980 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce041c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdfc15ced0 con 0x7fcdfc0ceb70 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce041c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0b4ce0 con 0x7fcdfc0b4370 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce01f65640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcdfc144980 0x7fcdfc15c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33514/0 (socket says 172.21.15.46:33514) 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce01f65640 1 -- 172.21.15.46:0/2706577354 learned_addr learned my addr 172.21.15.46:0/2706577354 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce01764640 1 --2- 172.21.15.46:0/2706577354 >> v2:172.21.15.121:3300/0 conn(0x7fcdfc0ceb70 0x7fcdfc0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce02766640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcdfc0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069264417 0 0) 0x7fcdfc071770 con 0x7fcdfc144980 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.196+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4003520 con 0x7fcdfc144980 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3983986820 0 0) 0x7fcde4003520 con 0x7fcdfc144980 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 msgr2=0x7fcdfc0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcdfc0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 >> v2:172.21.15.121:3300/0 conn(0x7fcdfc0ceb70 msgr2=0x7fcdfc0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 --2- 172.21.15.46:0/2706577354 >> v2:172.21.15.121:3300/0 conn(0x7fcdfc0ceb70 0x7fcdfc0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc0ccdc0 con 0x7fcdfc144980 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fcdf4001670 con 0x7fcdfc144980 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce02766640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcdfc0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 >> v1:172.21.15.46:6789/0 conn(0x7fcdfc144980 legacy=0x7fcdfc15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcde4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcde40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde4004d00 0x7fcde4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc15ced0 con 0x7fcde4003970 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0b4ce0 con 0x7fcde4004d00 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde400b980 con 0x7fcdfc0b4370 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01764640 1 --2- 172.21.15.46:0/2706577354 >> v2:172.21.15.121:3300/0 conn(0x7fcdfc0ceb70 0x7fcdfc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01f65640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcde4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce02766640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde4004d00 0x7fcde4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01764640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcde40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01f65640 1 -- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 msgr2=0x7fcde40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01f65640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc0b4370 0x7fcde40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01f65640 1 -- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde4004d00 msgr2=0x7fcde4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01f65640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde4004d00 0x7fcde4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.197+0000 7fce01f65640 1 -- 172.21.15.46:0/2706577354 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009a70 con 0x7fcde4003970 2023-12-16T21:14:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.198+0000 7fce01f65640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcde4003d90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c240 tx=0x7fcde40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a308c84ca2f4a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.198+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf402cb00 con 0x7fcde4003970 2023-12-16T21:14:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.198+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf402cc60 con 0x7fcde4003970 2023-12-16T21:14:15.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.198+0000 7fce00f63640 1 -- 172.21.15.46:0/2706577354 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4005af0 con 0x7fcde4003970 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce041c9640 1 -- 172.21.15.46:0/2706577354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 msgr2=0x7fcde4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce041c9640 1 --2- 172.21.15.46:0/2706577354 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcde4003d90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c240 tx=0x7fcde40057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce02766640 1 -- 172.21.15.46:0/2706577354 reap_dead start 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce041c9640 1 -- 172.21.15.46:0/2706577354 shutdown_connections 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce041c9640 1 -- 172.21.15.46:0/2706577354 >> 172.21.15.46:0/2706577354 conn(0x7fcdfc06a4d0 msgr2=0x7fcdfc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce041c9640 1 -- 172.21.15.46:0/2706577354 shutdown_connections 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.199+0000 7fce041c9640 1 -- 172.21.15.46:0/2706577354 wait complete. 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.200+0000 7fce041c9640 1 Processor -- start 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.200+0000 7fce041c9640 1 -- start start 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcdfc140b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdfc0c9310 0x7fcdfc0c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc15b670 0x7fcdfc15baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc0c44e0 con 0x7fcde4003970 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc15cb50 con 0x7fcdfc0c9310 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdfc15c030 con 0x7fcdfc15b670 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcdfc140b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcdfc140b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53378/0 (socket says 172.21.15.46:53378) 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 -- 172.21.15.46:0/4021118389 learned_addr learned my addr 172.21.15.46:0/4021118389 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 -- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc15b670 msgr2=0x7fcdfc15baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc15b670 0x7fcdfc15baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 -- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdfc0c9310 msgr2=0x7fcdfc0c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdfc0c9310 0x7fcdfc0c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 -- 172.21.15.46:0/4021118389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc0c66f0 con 0x7fcde4003970 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce01f65640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcdfc140b60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c360 tx=0x7fcdf4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf4039030 con 0x7fcde4003970 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf4004bf0 con 0x7fcde4003970 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdfc0c6130 con 0x7fcde4003970 2023-12-16T21:14:15.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf403c040 con 0x7fcde4003970 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.201+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdfc0c4100 con 0x7fcde4003970 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.202+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdfc0c42c0 con 0x7fcde4003970 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.203+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fcdf4032070 con 0x7fcde4003970 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.203+0000 7fcdfa7fc640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcde806ae40 0x7fcde806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.204+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdf40aa920 con 0x7fcde4003970 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.204+0000 7fce01764640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcde806ae40 0x7fcde806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.204+0000 7fce01764640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcde806ae40 0x7fcde806d300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0cb340 tx=0x7fcdf0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:15.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.208+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcdf4075a60 con 0x7fcde4003970 2023-12-16T21:14:15.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.383+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcdfc069400 con 0x7fcde4003970 2023-12-16T21:14:15.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.383+0000 7fcdfa7fc640 1 -- 172.21.15.46:0/4021118389 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcdf4079080 con 0x7fcde4003970 2023-12-16T21:14:15.383 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcde806ae40 msgr2=0x7fcde806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcde806ae40 0x7fcde806d300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0cb340 tx=0x7fcdf0008040 comp rx=0 tx=0).stop 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 msgr2=0x7fcdfc140b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcdfc140b60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcdf400c360 tx=0x7fcdf4001a00 comp rx=0 tx=0).stop 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 shutdown_connections 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcde806ae40 0x7fcde806d300 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcdfc15b670 0x7fcdfc15baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdfc0c9310 0x7fcdfc0c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 --2- 172.21.15.46:0/4021118389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcde4003970 0x7fcdfc140b60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 >> 172.21.15.46:0/4021118389 conn(0x7fcdfc06a4d0 msgr2=0x7fcdfc15aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 shutdown_connections 2023-12-16T21:14:15.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.385+0000 7fce041c9640 1 -- 172.21.15.46:0/4021118389 wait complete. 2023-12-16T21:14:15.396 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:15.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.503+0000 7f6873d78640 1 Processor -- start 2023-12-16T21:14:15.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.504+0000 7f6873d78640 1 -- start start 2023-12-16T21:14:15.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.504+0000 7f6873d78640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c15d330 0x7f686c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.504+0000 7f6873d78640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f686c0d2910 0x7f686c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.504+0000 7f6873d78640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686c0b2b10 con 0x7f686c0d2d90 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.504+0000 7f6873d78640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f686c0d1590 con 0x7f686c0d2910 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.504+0000 7f6873d78640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686c0d1870 con 0x7f686c15d330 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6871313640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f686c0d2910 0x7f686c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6872315640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f686c0d2d90 0x7f686c0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33524/0 (socket says 172.21.15.46:33524) 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6872315640 1 -- 172.21.15.46:0/2955050147 learned_addr learned my addr 172.21.15.46:0/2955050147 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6871b14640 1 --2- 172.21.15.46:0/2955050147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c15d330 0x7f686c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6871313640 1 -- 172.21.15.46:0/2955050147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c15d330 msgr2=0x7f686c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6871313640 1 --2- 172.21.15.46:0/2955050147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c15d330 0x7f686c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6871313640 1 -- 172.21.15.46:0/2955050147 >> v1:172.21.15.46:6789/0 conn(0x7f686c0d2d90 legacy=0x7f686c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6871313640 1 -- 172.21.15.46:0/2955050147 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c0d1b50 con 0x7f686c0d2910 2023-12-16T21:14:15.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.505+0000 7f6870b12640 1 -- 172.21.15.46:0/2955050147 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306791440 0 0) 0x7f686c0b2b10 con 0x7f686c0d2d90 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.506+0000 7f6871313640 1 --2- 172.21.15.46:0/2955050147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2910 0x7f686c0d31f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f685c0097a0 tx=0x7f685c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17776a7be4029591 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.506+0000 7f6870b12640 1 -- 172.21.15.46:0/2955050147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c02e070 con 0x7f686c0d2910 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.506+0000 7f6870b12640 1 -- 172.21.15.46:0/2955050147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c0315a0 con 0x7f686c0d2910 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.506+0000 7f6870b12640 1 -- 172.21.15.46:0/2955050147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c031930 con 0x7f686c0d2910 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.506+0000 7f6873d78640 1 -- 172.21.15.46:0/2955050147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2910 msgr2=0x7f686c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.506+0000 7f6873d78640 1 --2- 172.21.15.46:0/2955050147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2910 0x7f686c0d31f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f685c0097a0 tx=0x7f685c02b840 comp rx=0 tx=0).stop 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.507+0000 7f6873d78640 1 -- 172.21.15.46:0/2955050147 shutdown_connections 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.507+0000 7f6873d78640 1 --2- 172.21.15.46:0/2955050147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2910 0x7f686c0d31f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.507+0000 7f6873d78640 1 --2- 172.21.15.46:0/2955050147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c15d330 0x7f686c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.507+0000 7f6873d78640 1 -- 172.21.15.46:0/2955050147 >> 172.21.15.46:0/2955050147 conn(0x7f686c072020 msgr2=0x7f686c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:15.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.507+0000 7f6873d78640 1 -- 172.21.15.46:0/2955050147 shutdown_connections 2023-12-16T21:14:15.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.507+0000 7f6873d78640 1 -- 172.21.15.46:0/2955050147 wait complete. 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.508+0000 7f6873d78640 1 Processor -- start 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.508+0000 7f6873d78640 1 -- start start 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6873d78640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c0d2910 0x7f686c150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6873d78640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 0x7f686c0ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6873d78640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686c15d330 0x7f686c0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6873d78640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c150a70 con 0x7f686c0d2d90 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6873d78640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f686c0ca9c0 con 0x7f686c15d330 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6873d78640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686c0cb2e0 con 0x7f686c0d2910 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 0x7f686c0ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 0x7f686c0ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53394/0 (socket says 172.21.15.46:53394) 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 -- 172.21.15.46:0/3015054195 learned_addr learned my addr 172.21.15.46:0/3015054195 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871b14640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c0d2910 0x7f686c150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6872315640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686c15d330 0x7f686c0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 -- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c0d2910 msgr2=0x7f686c150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c0d2910 0x7f686c150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 -- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686c15d330 msgr2=0x7f686c0cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686c15d330 0x7f686c0cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.509+0000 7f6871313640 1 -- 172.21.15.46:0/3015054195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c009400 con 0x7f686c0d2d90 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.510+0000 7f6871313640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 0x7f686c0ca480 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f685c034040 tx=0x7f685c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.510+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c032840 con 0x7f686c0d2d90 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.510+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c0329c0 con 0x7f686c0d2d90 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.510+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f686c0c7a00 con 0x7f686c0d2d90 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.510+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c032d70 con 0x7f686c0d2d90 2023-12-16T21:14:15.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.510+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f686c0c7fc0 con 0x7f686c0d2d90 2023-12-16T21:14:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.511+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f686c0b47b0 con 0x7f686c0d2d90 2023-12-16T21:14:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.511+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f685c03b3f0 con 0x7f686c0d2d90 2023-12-16T21:14:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.512+0000 7f68627fc640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f685006ae40 0x7f685006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.512+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f685c0ab3f0 con 0x7f686c0d2d90 2023-12-16T21:14:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.512+0000 7f6871b14640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f685006ae40 0x7f685006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:15.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.513+0000 7f6871b14640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f685006ae40 0x7f685006d300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6868009b40 tx=0x7f6868009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:15.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.516+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f685c076550 con 0x7f686c0d2d90 2023-12-16T21:14:15.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.710+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f686c069400 con 0x7f686c0d2d90 2023-12-16T21:14:15.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.710+0000 7f68627fc640 1 -- 172.21.15.46:0/3015054195 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f685c079b70 con 0x7f686c0d2d90 2023-12-16T21:14:15.710 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f685006ae40 msgr2=0x7f685006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f685006ae40 0x7f685006d300 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6868009b40 tx=0x7f6868009190 comp rx=0 tx=0).stop 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 msgr2=0x7f686c0ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 0x7f686c0ca480 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f685c034040 tx=0x7f685c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 shutdown_connections 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f685006ae40 0x7f685006d300 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f686c0d2d90 0x7f686c0ca480 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f686c15d330 0x7f686c0cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 --2- 172.21.15.46:0/3015054195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f686c0d2910 0x7f686c150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 >> 172.21.15.46:0/3015054195 conn(0x7f686c072020 msgr2=0x7f686c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 shutdown_connections 2023-12-16T21:14:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:15.712+0000 7f6873d78640 1 -- 172.21.15.46:0/3015054195 wait complete. 2023-12-16T21:14:15.722 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2023-12-16T21:14:15.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:15.723 DEBUG:teuthology.orchestra.run.smithi121:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-12-16T21:14:15.756 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:15.756 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:14:15.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:15.756 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-16T21:14:16.112 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:16.112 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:16.130 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:16.131 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:16.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.339+0000 7f8369516640 1 Processor -- start 2023-12-16T21:14:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.339+0000 7f8369516640 1 -- start start 2023-12-16T21:14:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f8369516640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8364160fb0 0x7f8364161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f8369516640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83640d0ed0 0x7f83640d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f8369516640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8364161d20 con 0x7f836406ea50 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f8369516640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836415e760 con 0x7f83640d0ed0 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f8369516640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f836415d9a0 con 0x7f8364160fb0 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83640d0ed0 0x7f83640d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83640d0ed0 0x7f83640d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53398/0 (socket says 172.21.15.46:53398) 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 -- 172.21.15.46:0/875263877 learned_addr learned my addr 172.21.15.46:0/875263877 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f835bfff640 1 --2- 172.21.15.46:0/875263877 >> v2:172.21.15.121:3300/0 conn(0x7f8364160fb0 0x7f8364161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 -- 172.21.15.46:0/875263877 >> v2:172.21.15.121:3300/0 conn(0x7f8364160fb0 msgr2=0x7f8364161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 --2- 172.21.15.46:0/875263877 >> v2:172.21.15.121:3300/0 conn(0x7f8364160fb0 0x7f8364161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 -- 172.21.15.46:0/875263877 >> v1:172.21.15.121:6789/0 conn(0x7f836406ea50 legacy=0x7f836406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.340+0000 7f83637fe640 1 -- 172.21.15.46:0/875263877 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836415dc80 con 0x7f83640d0ed0 2023-12-16T21:14:16.341 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.341+0000 7f83637fe640 1 --2- 172.21.15.46:0/875263877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640d1320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f83640b6690 tx=0x7f835c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16721592de38599 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.341+0000 7f83627fc640 1 -- 172.21.15.46:0/875263877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c02ca80 con 0x7f83640d0ed0 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.341+0000 7f83627fc640 1 -- 172.21.15.46:0/875263877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f835c02cbe0 con 0x7f83640d0ed0 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.341+0000 7f83627fc640 1 -- 172.21.15.46:0/875263877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c0343f0 con 0x7f83640d0ed0 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.341+0000 7f8369516640 1 -- 172.21.15.46:0/875263877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 msgr2=0x7f83640d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.341+0000 7f8369516640 1 --2- 172.21.15.46:0/875263877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640d1320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f83640b6690 tx=0x7f835c02bee0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.342+0000 7f8369516640 1 -- 172.21.15.46:0/875263877 shutdown_connections 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.342+0000 7f8369516640 1 --2- 172.21.15.46:0/875263877 >> v2:172.21.15.121:3300/0 conn(0x7f8364160fb0 0x7f8364161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.342+0000 7f8369516640 1 --2- 172.21.15.46:0/875263877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640d1320 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.342+0000 7f8369516640 1 -- 172.21.15.46:0/875263877 >> 172.21.15.46:0/875263877 conn(0x7f836406a6b0 msgr2=0x7f836406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.342+0000 7f8369516640 1 -- 172.21.15.46:0/875263877 shutdown_connections 2023-12-16T21:14:16.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.342+0000 7f8369516640 1 -- 172.21.15.46:0/875263877 wait complete. 2023-12-16T21:14:16.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.343+0000 7f8369516640 1 Processor -- start 2023-12-16T21:14:16.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.343+0000 7f8369516640 1 -- start start 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.343+0000 7f8369516640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f836406ea50 0x7f836413bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8364160fb0 0x7f83640b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83640b42c0 con 0x7f83640d0ed0 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836413c100 con 0x7f8364160fb0 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83640b36b0 con 0x7f836406ea50 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53402/0 (socket says 172.21.15.46:53402) 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 -- 172.21.15.46:0/2593697301 learned_addr learned my addr 172.21.15.46:0/2593697301 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8362ffd640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f836406ea50 0x7f836413bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f83637fe640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8364160fb0 0x7f83640b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 -- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f836406ea50 msgr2=0x7f836413bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f836406ea50 0x7f836413bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 -- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8364160fb0 msgr2=0x7f83640b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8364160fb0 0x7f83640b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 -- 172.21.15.46:0/2593697301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c009b30 con 0x7f83640d0ed0 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835bfff640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640b4750 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f834c00d800 tx=0x7f834c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c00b6c0 con 0x7f83640d0ed0 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c00b820 con 0x7f83640d0ed0 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83640b3990 con 0x7f83640d0ed0 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c00bbb0 con 0x7f83640d0ed0 2023-12-16T21:14:16.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.344+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83640b3f50 con 0x7f83640d0ed0 2023-12-16T21:14:16.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.346+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83640b4430 con 0x7f83640d0ed0 2023-12-16T21:14:16.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.346+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f834c00bd10 con 0x7f83640d0ed0 2023-12-16T21:14:16.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.346+0000 7f835b7fe640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f834406ae40 0x7f834406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.346+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f834c089960 con 0x7f83640d0ed0 2023-12-16T21:14:16.347 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.347+0000 7f8362ffd640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f834406ae40 0x7f834406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.348 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.347+0000 7f8362ffd640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f834406ae40 0x7f834406d300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f836415d290 tx=0x7f8354009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.350+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f834c054b50 con 0x7f83640d0ed0 2023-12-16T21:14:16.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.531+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8364069400 con 0x7f83640d0ed0 2023-12-16T21:14:16.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.532+0000 7f835b7fe640 1 -- 172.21.15.46:0/2593697301 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f834c058170 con 0x7f83640d0ed0 2023-12-16T21:14:16.532 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:16.532 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":21,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:13:37.481612+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f834406ae40 msgr2=0x7f834406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f834406ae40 0x7f834406d300 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f836415d290 tx=0x7f8354009040 comp rx=0 tx=0).stop 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 msgr2=0x7f83640b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640b4750 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f834c00d800 tx=0x7f834c00dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 shutdown_connections 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f834406ae40 0x7f834406d300 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83640d0ed0 0x7f83640b4750 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8364160fb0 0x7f83640b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 --2- 172.21.15.46:0/2593697301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f836406ea50 0x7f836413bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 >> 172.21.15.46:0/2593697301 conn(0x7f836406a6b0 msgr2=0x7f836415f970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:16.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 shutdown_connections 2023-12-16T21:14:16.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.534+0000 7f8369516640 1 -- 172.21.15.46:0/2593697301 wait complete. 2023-12-16T21:14:16.545 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:16.545 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.657+0000 7ff19553e640 1 Processor -- start 2023-12-16T21:14:16.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.657+0000 7ff19553e640 1 -- start start 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.658+0000 7ff19553e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff190144980 0x7ff19015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.658+0000 7ff19553e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff190071770 con 0x7ff1900ceb70 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.658+0000 7ff19553e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1900ccb60 con 0x7ff1900b4370 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.658+0000 7ff19553e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1900ccdc0 con 0x7ff190144980 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff187fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff1900ceb70 0x7ff1900cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33540/0 (socket says 172.21.15.46:33540) 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff187fff640 1 -- 172.21.15.46:0/3719932855 learned_addr learned my addr 172.21.15.46:0/3719932855 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18effd640 1 --2- 172.21.15.46:0/3719932855 >> v2:172.21.15.121:3300/0 conn(0x7ff190144980 0x7ff19015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18e7fc640 1 -- 172.21.15.46:0/3719932855 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947725935 0 0) 0x7ff190071770 con 0x7ff1900ceb70 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18e7fc640 1 -- 172.21.15.46:0/3719932855 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff17c003520 con 0x7ff1900ceb70 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18e7fc640 1 -- 172.21.15.46:0/3719932855 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3345638496 0 0) 0x7ff17c003520 con 0x7ff1900ceb70 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18effd640 1 -- 172.21.15.46:0/3719932855 >> v1:172.21.15.121:6789/0 conn(0x7ff1900b4370 legacy=0x7ff190071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18effd640 1 -- 172.21.15.46:0/3719932855 >> v1:172.21.15.46:6789/0 conn(0x7ff1900ceb70 legacy=0x7ff1900cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.659+0000 7ff18effd640 1 -- 172.21.15.46:0/3719932855 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900cd0a0 con 0x7ff190144980 2023-12-16T21:14:16.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.660+0000 7ff18effd640 1 --2- 172.21.15.46:0/3719932855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff190144980 0x7ff19015c7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff1800097a0 tx=0x7ff18002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c067843496966a86 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.662+0000 7ff18e7fc640 1 -- 172.21.15.46:0/3719932855 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18002f050 con 0x7ff190144980 2023-12-16T21:14:16.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.662+0000 7ff18e7fc640 1 -- 172.21.15.46:0/3719932855 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff180031500 con 0x7ff190144980 2023-12-16T21:14:16.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.662+0000 7ff18e7fc640 1 -- 172.21.15.46:0/3719932855 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178001670 con 0x7ff190144980 2023-12-16T21:14:16.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 -- 172.21.15.46:0/3719932855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff190144980 msgr2=0x7ff19015c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 --2- 172.21.15.46:0/3719932855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff190144980 0x7ff19015c7c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff1800097a0 tx=0x7ff18002b820 comp rx=0 tx=0).stop 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 -- 172.21.15.46:0/3719932855 shutdown_connections 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 --2- 172.21.15.46:0/3719932855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff190144980 0x7ff19015c7c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 -- 172.21.15.46:0/3719932855 >> 172.21.15.46:0/3719932855 conn(0x7ff19006a4d0 msgr2=0x7ff19006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 -- 172.21.15.46:0/3719932855 shutdown_connections 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.663+0000 7ff19553e640 1 -- 172.21.15.46:0/3719932855 wait complete. 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.664+0000 7ff19553e640 1 Processor -- start 2023-12-16T21:14:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.664+0000 7ff19553e640 1 -- start start 2023-12-16T21:14:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff19553e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1900b4370 0x7ff1901578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff19553e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 0x7ff190139820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff19553e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff190144980 0x7ff1900af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff19553e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190138350 con 0x7ff1900ceb70 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff19553e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190138780 con 0x7ff1900b4370 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff19553e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff190138a60 con 0x7ff190144980 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 0x7ff190139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 0x7ff190139820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53416/0 (socket says 172.21.15.46:53416) 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 -- 172.21.15.46:0/1911500103 learned_addr learned my addr 172.21.15.46:0/1911500103 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff18effd640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1900b4370 0x7ff1901578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff18f7fe640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff190144980 0x7ff1900af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 -- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff190144980 msgr2=0x7ff1900af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff190144980 0x7ff1900af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 -- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1900b4370 msgr2=0x7ff1901578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1900b4370 0x7ff1901578c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 -- 172.21.15.46:0/1911500103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180009400 con 0x7ff1900ceb70 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff18f7fe640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff190144980 0x7ff1900af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.665+0000 7ff187fff640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 0x7ff190139820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff17800c730 tx=0x7ff17800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.666+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178005ad0 con 0x7ff1900ceb70 2023-12-16T21:14:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.666+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff178005c50 con 0x7ff1900ceb70 2023-12-16T21:14:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.666+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178007070 con 0x7ff1900ceb70 2023-12-16T21:14:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.666+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff190138d40 con 0x7ff1900ceb70 2023-12-16T21:14:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.666+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff190139300 con 0x7ff1900ceb70 2023-12-16T21:14:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.667+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff190139520 con 0x7ff1900ceb70 2023-12-16T21:14:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.667+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff1780071d0 con 0x7ff1900ceb70 2023-12-16T21:14:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.668+0000 7ff1877fe640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff17006ae40 0x7ff17006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.668+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff178088d90 con 0x7ff1900ceb70 2023-12-16T21:14:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.668+0000 7ff18effd640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff17006ae40 0x7ff17006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.668+0000 7ff18effd640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff17006ae40 0x7ff17006d300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff180035000 tx=0x7ff180008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.672+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff178053e70 con 0x7ff1900ceb70 2023-12-16T21:14:16.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.847+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff19015cc80 con 0x7ff1900ceb70 2023-12-16T21:14:16.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.847+0000 7ff1877fe640 1 -- 172.21.15.46:0/1911500103 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff178017020 con 0x7ff1900ceb70 2023-12-16T21:14:16.847 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff17006ae40 msgr2=0x7ff17006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff17006ae40 0x7ff17006d300 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff180035000 tx=0x7ff180008030 comp rx=0 tx=0).stop 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 msgr2=0x7ff190139820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 0x7ff190139820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff17800c730 tx=0x7ff17800cbf0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 shutdown_connections 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff17006ae40 0x7ff17006d300 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.849 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1900ceb70 0x7ff190139820 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff190144980 0x7ff1900af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 --2- 172.21.15.46:0/1911500103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1900b4370 0x7ff1901578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 >> 172.21.15.46:0/1911500103 conn(0x7ff19006a4d0 msgr2=0x7ff1900b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:16.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 shutdown_connections 2023-12-16T21:14:16.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.849+0000 7ff19553e640 1 -- 172.21.15.46:0/1911500103 wait complete. 2023-12-16T21:14:16.860 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:16.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.969+0000 7f9c4fdd5640 1 Processor -- start 2023-12-16T21:14:16.969 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.969+0000 7f9c4fdd5640 1 -- start start 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4fdd5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c48150560 0x7f9c480b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4fdd5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9c48150940 0x7f9c480ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4fdd5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480b49f0 con 0x7f9c480b3390 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4fdd5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480b3de0 con 0x7f9c48150560 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4fdd5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c480b40c0 con 0x7f9c48150940 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c48150560 0x7f9c480b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9c48150560 0x7f9c480b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53432/0 (socket says 172.21.15.46:53432) 2023-12-16T21:14:16.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 -- 172.21.15.46:0/1416459818 learned_addr learned my addr 172.21.15.46:0/1416459818 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4e372640 1 --2- 172.21.15.46:0/1416459818 >> v2:172.21.15.121:3300/0 conn(0x7f9c48150940 0x7f9c480ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 -- 172.21.15.46:0/1416459818 >> v2:172.21.15.121:3300/0 conn(0x7f9c48150940 msgr2=0x7f9c480ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 --2- 172.21.15.46:0/1416459818 >> v2:172.21.15.121:3300/0 conn(0x7f9c48150940 0x7f9c480ce0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 -- 172.21.15.46:0/1416459818 >> v1:172.21.15.121:6789/0 conn(0x7f9c480b3390 legacy=0x7f9c4815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:16.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.970+0000 7f9c4db71640 1 -- 172.21.15.46:0/1416459818 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c480b43a0 con 0x7f9c48150560 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.971+0000 7f9c4db71640 1 --2- 172.21.15.46:0/1416459818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c48150560 0x7f9c480b2e50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c4815d160 tx=0x7f9c4002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=470ecc041cae54a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.971+0000 7f9c4cb6f640 1 -- 172.21.15.46:0/1416459818 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40031870 con 0x7f9c48150560 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.971+0000 7f9c4cb6f640 1 -- 172.21.15.46:0/1416459818 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c400319d0 con 0x7f9c48150560 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.971+0000 7f9c4cb6f640 1 -- 172.21.15.46:0/1416459818 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40031d60 con 0x7f9c48150560 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/1416459818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c48150560 msgr2=0x7f9c480b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/1416459818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c48150560 0x7f9c480b2e50 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9c4815d160 tx=0x7f9c4002b910 comp rx=0 tx=0).stop 2023-12-16T21:14:16.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/1416459818 shutdown_connections 2023-12-16T21:14:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/1416459818 >> v2:172.21.15.121:3300/0 conn(0x7f9c48150940 0x7f9c480ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/1416459818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c48150560 0x7f9c480b2e50 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/1416459818 >> 172.21.15.46:0/1416459818 conn(0x7f9c4806a4d0 msgr2=0x7f9c4806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/1416459818 shutdown_connections 2023-12-16T21:14:16.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.972+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/1416459818 wait complete. 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.973+0000 7f9c4fdd5640 1 Processor -- start 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.973+0000 7f9c4fdd5640 1 -- start start 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4fdd5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 0x7f9c48148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4fdd5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c48150560 0x7f9c48155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4fdd5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c48147e50 0x7f9c4814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4fdd5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48149870 con 0x7f9c480b3390 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4fdd5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48148bf0 con 0x7f9c48147e50 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4fdd5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48148ed0 con 0x7f9c48150560 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 0x7f9c48148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 0x7f9c48148610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53436/0 (socket says 172.21.15.46:53436) 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 -- 172.21.15.46:0/2559239061 learned_addr learned my addr 172.21.15.46:0/2559239061 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4d370640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c48150560 0x7f9c48155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4e372640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c48147e50 0x7f9c4814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 -- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c48150560 msgr2=0x7f9c48155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c48150560 0x7f9c48155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 -- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c48147e50 msgr2=0x7f9c4814e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c48147e50 0x7f9c4814e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4db71640 1 -- 172.21.15.46:0/2559239061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c400094d0 con 0x7f9c480b3390 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.974+0000 7f9c4d370640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c48150560 0x7f9c48155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.975+0000 7f9c4db71640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 0x7f9c48148610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9c40000c00 tx=0x7f9c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.975+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c40031840 con 0x7f9c480b3390 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.975+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c400443f0 con 0x7f9c480b3390 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.975+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c400447a0 con 0x7f9c480b3390 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.975+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c481491b0 con 0x7f9c480b3390 2023-12-16T21:14:16.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.975+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c48156290 con 0x7f9c480b3390 2023-12-16T21:14:16.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.976+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c481499b0 con 0x7f9c480b3390 2023-12-16T21:14:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.976+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9c4004d450 con 0x7f9c480b3390 2023-12-16T21:14:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.977+0000 7f9c467fc640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9c3406ae40 0x7f9c3406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.977+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9c4002d040 con 0x7f9c480b3390 2023-12-16T21:14:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.977+0000 7f9c4d370640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9c3406ae40 0x7f9c3406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:16.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.977+0000 7f9c4d370640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9c3406ae40 0x7f9c3406d300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c48156970 tx=0x7f9c3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:16.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:16.981+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9c481499b0 con 0x7f9c480b3390 2023-12-16T21:14:17.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.155+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c481499b0 con 0x7f9c480b3390 2023-12-16T21:14:17.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.156+0000 7f9c467fc640 1 -- 172.21.15.46:0/2559239061 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c40035020 con 0x7f9c480b3390 2023-12-16T21:14:17.156 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:17.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9c3406ae40 msgr2=0x7f9c3406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9c3406ae40 0x7f9c3406d300 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9c48156970 tx=0x7f9c3c009040 comp rx=0 tx=0).stop 2023-12-16T21:14:17.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 msgr2=0x7f9c48148610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 0x7f9c48148610 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9c40000c00 tx=0x7f9c400046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 shutdown_connections 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9c3406ae40 0x7f9c3406d300 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c48147e50 0x7f9c4814e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c48150560 0x7f9c48155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 --2- 172.21.15.46:0/2559239061 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9c480b3390 0x7f9c48148610 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 >> 172.21.15.46:0/2559239061 conn(0x7f9c4806a4d0 msgr2=0x7f9c4814fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:17.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 shutdown_connections 2023-12-16T21:14:17.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.158+0000 7f9c4fdd5640 1 -- 172.21.15.46:0/2559239061 wait complete. 2023-12-16T21:14:17.168 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:17.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.278+0000 7f5677651640 1 Processor -- start 2023-12-16T21:14:17.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.278+0000 7f5677651640 1 -- start start 2023-12-16T21:14:17.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.278+0000 7f5677651640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5670140310 0x7f56701618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5677651640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f56701409d0 0x7f567006a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5677651640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56701428d0 0x7f567006aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5677651640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5670161e20 con 0x7f56701428d0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5677651640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56701406f0 con 0x7f56701409d0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5677651640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5670143960 con 0x7f5670140310 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56701428d0 0x7f567006aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f56701428d0 0x7f567006aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53442/0 (socket says 172.21.15.46:53442) 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 -- 172.21.15.46:0/2326946358 learned_addr learned my addr 172.21.15.46:0/2326946358 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f56753ed640 1 --2- 172.21.15.46:0/2326946358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5670140310 0x7f56701618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5674bec640 1 --2- 172.21.15.46:0/2326946358 >> v2:172.21.15.121:3300/0 conn(0x7f56701409d0 0x7f567006a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 -- 172.21.15.46:0/2326946358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5670140310 msgr2=0x7f56701618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 --2- 172.21.15.46:0/2326946358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5670140310 0x7f56701618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 -- 172.21.15.46:0/2326946358 >> v2:172.21.15.121:3300/0 conn(0x7f56701409d0 msgr2=0x7f567006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 --2- 172.21.15.46:0/2326946358 >> v2:172.21.15.121:3300/0 conn(0x7f56701409d0 0x7f567006a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.279+0000 7f5675bee640 1 -- 172.21.15.46:0/2326946358 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5670142de0 con 0x7f56701428d0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.280+0000 7f5675bee640 1 --2- 172.21.15.46:0/2326946358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701428d0 0x7f567006aa10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f567006e630 tx=0x7f566002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4203299433320b17 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.280 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.280+0000 7f566ffff640 1 -- 172.21.15.46:0/2326946358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566002c9e0 con 0x7f56701428d0 2023-12-16T21:14:17.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.280+0000 7f566ffff640 1 -- 172.21.15.46:0/2326946358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566002cb40 con 0x7f56701428d0 2023-12-16T21:14:17.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.280+0000 7f566ffff640 1 -- 172.21.15.46:0/2326946358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56600343f0 con 0x7f56701428d0 2023-12-16T21:14:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 -- 172.21.15.46:0/2326946358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701428d0 msgr2=0x7f567006aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 --2- 172.21.15.46:0/2326946358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701428d0 0x7f567006aa10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f567006e630 tx=0x7f566002be40 comp rx=0 tx=0).stop 2023-12-16T21:14:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 -- 172.21.15.46:0/2326946358 shutdown_connections 2023-12-16T21:14:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 --2- 172.21.15.46:0/2326946358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56701428d0 0x7f567006aa10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 --2- 172.21.15.46:0/2326946358 >> v2:172.21.15.121:3300/0 conn(0x7f56701409d0 0x7f567006a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 --2- 172.21.15.46:0/2326946358 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5670140310 0x7f56701618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 -- 172.21.15.46:0/2326946358 >> 172.21.15.46:0/2326946358 conn(0x7f56700b1e50 msgr2=0x7f56700b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 -- 172.21.15.46:0/2326946358 shutdown_connections 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.281+0000 7f5677651640 1 -- 172.21.15.46:0/2326946358 wait complete. 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.282+0000 7f5677651640 1 Processor -- start 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.282+0000 7f5677651640 1 -- start start 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 0x7f56700b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56701409d0 0x7f567013da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56701428d0 0x7f56700ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56700b51e0 con 0x7f5670140310 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567015c8e0 con 0x7f56701409d0 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56701440a0 con 0x7f56701428d0 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 0x7f56700b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 0x7f56700b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53458/0 (socket says 172.21.15.46:53458) 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 -- 172.21.15.46:0/2647996161 learned_addr learned my addr 172.21.15.46:0/2647996161 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5675bee640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56701428d0 0x7f56700ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5674bec640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56701409d0 0x7f567013da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 -- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56701428d0 msgr2=0x7f56700ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56701428d0 0x7f56700ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 -- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56701409d0 msgr2=0x7f567013da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56701409d0 0x7f567013da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 -- 172.21.15.46:0/2647996161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5660009b30 con 0x7f5670140310 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5675bee640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56701428d0 0x7f56700ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f56753ed640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 0x7f56700b4ca0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f566800d5f0 tx=0x7f566800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668014620 con 0x7f5670140310 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5668014780 con 0x7f5670140310 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56700b3950 con 0x7f5670140310 2023-12-16T21:14:17.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.283+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668014b30 con 0x7f5670140310 2023-12-16T21:14:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.284+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56700b3c30 con 0x7f5670140310 2023-12-16T21:14:17.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.285+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567013dfa0 con 0x7f5670140310 2023-12-16T21:14:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.285+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5668014c90 con 0x7f5670140310 2023-12-16T21:14:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.285+0000 7f566dffb640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f565c06ae40 0x7f565c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.286+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5668089d10 con 0x7f5670140310 2023-12-16T21:14:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.286+0000 7f5674bec640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f565c06ae40 0x7f565c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.286+0000 7f5674bec640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f565c06ae40 0x7f565c06d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5664006fd0 tx=0x7f5664008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.290+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5668054dd0 con 0x7f5670140310 2023-12-16T21:14:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.464+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5670069400 con 0x7f5670140310 2023-12-16T21:14:17.465 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.465+0000 7f566dffb640 1 -- 172.21.15.46:0/2647996161 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56680180a0 con 0x7f5670140310 2023-12-16T21:14:17.465 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f565c06ae40 msgr2=0x7f565c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f565c06ae40 0x7f565c06d300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5664006fd0 tx=0x7f5664008040 comp rx=0 tx=0).stop 2023-12-16T21:14:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 msgr2=0x7f56700b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 0x7f56700b4ca0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f566800d5f0 tx=0x7f566800dab0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 shutdown_connections 2023-12-16T21:14:17.467 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f565c06ae40 0x7f565c06d300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56701428d0 0x7f56700ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56701409d0 0x7f567013da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 --2- 172.21.15.46:0/2647996161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5670140310 0x7f56700b4ca0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 >> 172.21.15.46:0/2647996161 conn(0x7f56700b1e50 msgr2=0x7f567013ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:17.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 shutdown_connections 2023-12-16T21:14:17.468 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.467+0000 7f5677651640 1 -- 172.21.15.46:0/2647996161 wait complete. 2023-12-16T21:14:17.478 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.586+0000 7f92fb2bb640 1 Processor -- start 2023-12-16T21:14:17.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.586+0000 7f92fb2bb640 1 -- start start 2023-12-16T21:14:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92fb2bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f41618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92fb2bb640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92f41409d0 0x7f92f406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92fb2bb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f41406f0 con 0x7f92f4140310 2023-12-16T21:14:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92fb2bb640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f4161e20 con 0x7f92f41409d0 2023-12-16T21:14:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92fb2bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4143960 con 0x7f92f406abe0 2023-12-16T21:14:17.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92f9057640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f92f4140310 0x7f92f406a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33544/0 (socket says 172.21.15.46:33544) 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92f9057640 1 -- 172.21.15.46:0/3669495439 learned_addr learned my addr 172.21.15.46:0/3669495439 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92f8856640 1 --2- 172.21.15.46:0/3669495439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f41618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92f9858640 1 --2- 172.21.15.46:0/3669495439 >> v2:172.21.15.121:3300/0 conn(0x7f92f41409d0 0x7f92f406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92f3fff640 1 -- 172.21.15.46:0/3669495439 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670217597 0 0) 0x7f92f41406f0 con 0x7f92f4140310 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.587+0000 7f92f3fff640 1 -- 172.21.15.46:0/3669495439 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92dc003520 con 0x7f92f4140310 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.588+0000 7f92f8856640 1 -- 172.21.15.46:0/3669495439 >> v2:172.21.15.121:3300/0 conn(0x7f92f41409d0 msgr2=0x7f92f406efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.588+0000 7f92f8856640 1 --2- 172.21.15.46:0/3669495439 >> v2:172.21.15.121:3300/0 conn(0x7f92f41409d0 0x7f92f406efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.588+0000 7f92f8856640 1 -- 172.21.15.46:0/3669495439 >> v1:172.21.15.46:6789/0 conn(0x7f92f4140310 legacy=0x7f92f406a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.588+0000 7f92f8856640 1 -- 172.21.15.46:0/3669495439 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4142de0 con 0x7f92f406abe0 2023-12-16T21:14:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.588+0000 7f92f8856640 1 --2- 172.21.15.46:0/3669495439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f41618e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92e80097a0 tx=0x7f92e802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d9eee924407026b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92f3fff640 1 -- 172.21.15.46:0/3669495439 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e802e070 con 0x7f92f406abe0 2023-12-16T21:14:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92f3fff640 1 -- 172.21.15.46:0/3669495439 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e80315a0 con 0x7f92f406abe0 2023-12-16T21:14:17.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92f3fff640 1 -- 172.21.15.46:0/3669495439 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8031910 con 0x7f92f406abe0 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 -- 172.21.15.46:0/3669495439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 msgr2=0x7f92f41618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/3669495439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f41618e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f92e80097a0 tx=0x7f92e802b840 comp rx=0 tx=0).stop 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 -- 172.21.15.46:0/3669495439 shutdown_connections 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/3669495439 >> v2:172.21.15.121:3300/0 conn(0x7f92f41409d0 0x7f92f406efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/3669495439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f41618e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 -- 172.21.15.46:0/3669495439 >> 172.21.15.46:0/3669495439 conn(0x7f92f40b1e50 msgr2=0x7f92f40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 -- 172.21.15.46:0/3669495439 shutdown_connections 2023-12-16T21:14:17.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.589+0000 7f92fb2bb640 1 -- 172.21.15.46:0/3669495439 wait complete. 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.590+0000 7f92fb2bb640 1 Processor -- start 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 -- start start 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f4144540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92f4140310 0x7f92f40aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 0x7f92f41571a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4145850 con 0x7f92f41409d0 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f40af2c0 con 0x7f92f4140310 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92fb2bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f4144bd0 con 0x7f92f406abe0 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92f9858640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 0x7f92f41571a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92f9858640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 0x7f92f41571a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53460/0 (socket says 172.21.15.46:53460) 2023-12-16T21:14:17.592 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.591+0000 7f92f9858640 1 -- 172.21.15.46:0/2154386642 learned_addr learned my addr 172.21.15.46:0/2154386642 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9057640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f4144540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f8856640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92f4140310 0x7f92f40aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9858640 1 -- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 msgr2=0x7f92f4144540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9858640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f4144540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9858640 1 -- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92f4140310 msgr2=0x7f92f40aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9858640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92f4140310 0x7f92f40aed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9858640 1 -- 172.21.15.46:0/2154386642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8009400 con 0x7f92f41409d0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f9858640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 0x7f92f41571a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f92e400efc0 tx=0x7f92e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4009ca0 con 0x7f92f41409d0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e400f040 con 0x7f92f41409d0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f4144eb0 con 0x7f92f41409d0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40146c0 con 0x7f92f41409d0 2023-12-16T21:14:17.593 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.592+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f4145470 con 0x7f92f41409d0 2023-12-16T21:14:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.593+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f4145990 con 0x7f92f41409d0 2023-12-16T21:14:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.594+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f92e4014820 con 0x7f92f41409d0 2023-12-16T21:14:17.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.594+0000 7f92f1ffb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92e006ae40 0x7f92e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.594+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f92e408a600 con 0x7f92f41409d0 2023-12-16T21:14:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.594+0000 7f92f9057640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92e006ae40 0x7f92e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.595+0000 7f92f9057640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92e006ae40 0x7f92e006d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f92ec005880 tx=0x7f92ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.598+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f92f4145990 con 0x7f92f41409d0 2023-12-16T21:14:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.794+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f92f4145990 con 0x7f92f41409d0 2023-12-16T21:14:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.794+0000 7f92f1ffb640 1 -- 172.21.15.46:0/2154386642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f92f4145990 con 0x7f92f41409d0 2023-12-16T21:14:17.794 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92e006ae40 msgr2=0x7f92e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92e006ae40 0x7f92e006d300 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f92ec005880 tx=0x7f92ec009040 comp rx=0 tx=0).stop 2023-12-16T21:14:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 msgr2=0x7f92f41571a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 0x7f92f41571a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f92e400efc0 tx=0x7f92e400c490 comp rx=0 tx=0).stop 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 shutdown_connections 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f92e006ae40 0x7f92e006d300 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f92f41409d0 0x7f92f41571a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92f4140310 0x7f92f40aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 --2- 172.21.15.46:0/2154386642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92f406abe0 0x7f92f4144540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.796+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 >> 172.21.15.46:0/2154386642 conn(0x7f92f40b1e50 msgr2=0x7f92f40afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.797+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 shutdown_connections 2023-12-16T21:14:17.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.797+0000 7f92fb2bb640 1 -- 172.21.15.46:0/2154386642 wait complete. 2023-12-16T21:14:17.807 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:17.807 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:17.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:17.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.968+0000 7fa6851a9640 1 Processor -- start 2023-12-16T21:14:17.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.968+0000 7fa6851a9640 1 -- start start 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa6851a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa68006ea50 0x7fa68006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa6851a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6800d0ed0 0x7fa6800d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa6851a9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680161d20 con 0x7fa680160fb0 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa6851a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68006f370 con 0x7fa6800d0ed0 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa6851a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68015e760 con 0x7fa68006ea50 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa67f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6800d0ed0 0x7fa6800d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa67f59e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa6800d0ed0 0x7fa6800d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53474/0 (socket says 172.21.15.46:53474) 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa67f59e640 1 -- 172.21.15.46:0/4076992484 learned_addr learned my addr 172.21.15.46:0/4076992484 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.969+0000 7fa67e59c640 1 --1- 172.21.15.46:0/4076992484 >> v1:172.21.15.46:6789/0 conn(0x7fa680160fb0 0x7fa680161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33550/0 (socket says 172.21.15.46:33550) 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.970+0000 7fa67f59e640 1 -- 172.21.15.46:0/4076992484 >> v2:172.21.15.121:3300/0 conn(0x7fa68006ea50 msgr2=0x7fa68006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.970+0000 7fa67f59e640 1 --2- 172.21.15.46:0/4076992484 >> v2:172.21.15.121:3300/0 conn(0x7fa68006ea50 0x7fa68006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.970+0000 7fa67f59e640 1 -- 172.21.15.46:0/4076992484 >> v1:172.21.15.46:6789/0 conn(0x7fa680160fb0 legacy=0x7fa680161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.970+0000 7fa67f59e640 1 -- 172.21.15.46:0/4076992484 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68015d9a0 con 0x7fa6800d0ed0 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.970+0000 7fa67dd9b640 1 -- 172.21.15.46:0/4076992484 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744041418 0 0) 0x7fa680161d20 con 0x7fa680160fb0 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.970+0000 7fa67f59e640 1 --2- 172.21.15.46:0/4076992484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6800d0ed0 0x7fa6800d1320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa68006f680 tx=0x7fa67802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f4e718110c32a3c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa67dd9b640 1 -- 172.21.15.46:0/4076992484 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67802ca80 con 0x7fa6800d0ed0 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa67dd9b640 1 -- 172.21.15.46:0/4076992484 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67802cbe0 con 0x7fa6800d0ed0 2023-12-16T21:14:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa67dd9b640 1 -- 172.21.15.46:0/4076992484 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6780343f0 con 0x7fa6800d0ed0 2023-12-16T21:14:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 -- 172.21.15.46:0/4076992484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6800d0ed0 msgr2=0x7fa6800d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 --2- 172.21.15.46:0/4076992484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6800d0ed0 0x7fa6800d1320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa68006f680 tx=0x7fa67802bee0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 -- 172.21.15.46:0/4076992484 shutdown_connections 2023-12-16T21:14:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 --2- 172.21.15.46:0/4076992484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa6800d0ed0 0x7fa6800d1320 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 --2- 172.21.15.46:0/4076992484 >> v2:172.21.15.121:3300/0 conn(0x7fa68006ea50 0x7fa68006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 -- 172.21.15.46:0/4076992484 >> 172.21.15.46:0/4076992484 conn(0x7fa68006a6b0 msgr2=0x7fa68006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 -- 172.21.15.46:0/4076992484 shutdown_connections 2023-12-16T21:14:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.971+0000 7fa6851a9640 1 -- 172.21.15.46:0/4076992484 wait complete. 2023-12-16T21:14:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.973+0000 7fa6851a9640 1 Processor -- start 2023-12-16T21:14:17.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.973+0000 7fa6851a9640 1 -- start start 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6851a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa68006ea50 0x7fa68015ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6851a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6800d0ed0 0x7fa680139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6851a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 0x7fa68013ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6851a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68015fcb0 con 0x7fa680160fb0 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6851a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6801583a0 con 0x7fa68006ea50 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6851a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa680158680 con 0x7fa6800d0ed0 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 0x7fa68013ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 0x7fa68013ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53488/0 (socket says 172.21.15.46:53488) 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 -- 172.21.15.46:0/2260346393 learned_addr learned my addr 172.21.15.46:0/2260346393 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67ed9d640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa68006ea50 0x7fa68015ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67e59c640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6800d0ed0 0x7fa680139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 -- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6800d0ed0 msgr2=0x7fa680139aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6800d0ed0 0x7fa680139aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 -- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa68006ea50 msgr2=0x7fa68015ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa68006ea50 0x7fa68015ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 -- 172.21.15.46:0/2260346393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa678009b30 con 0x7fa680160fb0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa67f59e640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 0x7fa68013ee90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa678036040 tx=0x7fa678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.974+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa678009510 con 0x7fa680160fb0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.975+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67803e450 con 0x7fa680160fb0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.975+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa680158960 con 0x7fa680160fb0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.975+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67803e7e0 con 0x7fa680160fb0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.975+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa680158f20 con 0x7fa680160fb0 2023-12-16T21:14:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.976+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa680159140 con 0x7fa680160fb0 2023-12-16T21:14:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.976+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa67802cd50 con 0x7fa680160fb0 2023-12-16T21:14:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.976+0000 7fa6677fe640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa66c06ae40 0x7fa66c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.977+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa6780ac140 con 0x7fa680160fb0 2023-12-16T21:14:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.977+0000 7fa67ed9d640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa66c06ae40 0x7fa66c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.977+0000 7fa67ed9d640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa66c06ae40 0x7fa66c06d300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6700098e0 tx=0x7fa670009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:17.981+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa6780772b0 con 0x7fa680160fb0 2023-12-16T21:14:18.160 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.160+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa68004ed60 con 0x7fa680160fb0 2023-12-16T21:14:18.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.161+0000 7fa6677fe640 1 -- 172.21.15.46:0/2260346393 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16754 (secure 0 0 0) 0x7fa67807a8d0 con 0x7fa680160fb0 2023-12-16T21:14:18.161 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:18.162 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:17.654454+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa66c06ae40 msgr2=0x7fa66c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa66c06ae40 0x7fa66c06d300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6700098e0 tx=0x7fa670009060 comp rx=0 tx=0).stop 2023-12-16T21:14:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 msgr2=0x7fa68013ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 0x7fa68013ee90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa678036040 tx=0x7fa678004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:18.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 shutdown_connections 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa66c06ae40 0x7fa66c06d300 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa680160fb0 0x7fa68013ee90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6800d0ed0 0x7fa680139aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 --2- 172.21.15.46:0/2260346393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa68006ea50 0x7fa68015ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 >> 172.21.15.46:0/2260346393 conn(0x7fa68006a6b0 msgr2=0x7fa6801416d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 shutdown_connections 2023-12-16T21:14:18.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.163+0000 7fa6851a9640 1 -- 172.21.15.46:0/2260346393 wait complete. 2023-12-16T21:14:18.174 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:18.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:18.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.285+0000 7fa28c89b640 1 Processor -- start 2023-12-16T21:14:18.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.285+0000 7fa28c89b640 1 -- start start 2023-12-16T21:14:18.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28c89b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa28806abe0 0x7fa2881618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28c89b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2881409d0 0x7fa28806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28c89b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2881406f0 con 0x7fa288140310 2023-12-16T21:14:18.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28c89b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa288161e20 con 0x7fa28806abe0 2023-12-16T21:14:18.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28c89b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa288143960 con 0x7fa2881409d0 2023-12-16T21:14:18.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa286d9d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2881409d0 0x7fa28806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa27ffff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa28806abe0 0x7fa2881618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28659c640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa288140310 0x7fa28806a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57892/0 (socket says 172.21.15.46:57892) 2023-12-16T21:14:18.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa28659c640 1 -- 172.21.15.46:0/3019599374 learned_addr learned my addr 172.21.15.46:0/3019599374 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:18.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa285d9b640 1 -- 172.21.15.46:0/3019599374 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341006410 0 0) 0x7fa2881406f0 con 0x7fa288140310 2023-12-16T21:14:18.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa285d9b640 1 -- 172.21.15.46:0/3019599374 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa274003520 con 0x7fa288140310 2023-12-16T21:14:18.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa286d9d640 1 -- 172.21.15.46:0/3019599374 >> v2:172.21.15.121:3300/0 conn(0x7fa28806abe0 msgr2=0x7fa2881618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:18.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa286d9d640 1 --2- 172.21.15.46:0/3019599374 >> v2:172.21.15.121:3300/0 conn(0x7fa28806abe0 0x7fa2881618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa286d9d640 1 -- 172.21.15.46:0/3019599374 >> v1:172.21.15.121:6789/0 conn(0x7fa288140310 legacy=0x7fa28806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.286+0000 7fa286d9d640 1 -- 172.21.15.46:0/3019599374 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288142de0 con 0x7fa2881409d0 2023-12-16T21:14:18.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.287+0000 7fa286d9d640 1 --2- 172.21.15.46:0/3019599374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2881409d0 0x7fa28806efb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa280009d90 tx=0x7fa28002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e611a9d18d8bc7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.287+0000 7fa285d9b640 1 -- 172.21.15.46:0/3019599374 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28002cbc0 con 0x7fa2881409d0 2023-12-16T21:14:18.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.287+0000 7fa285d9b640 1 -- 172.21.15.46:0/3019599374 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28002cd20 con 0x7fa2881409d0 2023-12-16T21:14:18.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.287+0000 7fa285d9b640 1 -- 172.21.15.46:0/3019599374 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280034660 con 0x7fa2881409d0 2023-12-16T21:14:18.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 -- 172.21.15.46:0/3019599374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2881409d0 msgr2=0x7fa28806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 --2- 172.21.15.46:0/3019599374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2881409d0 0x7fa28806efb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa280009d90 tx=0x7fa28002be10 comp rx=0 tx=0).stop 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 -- 172.21.15.46:0/3019599374 shutdown_connections 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 --2- 172.21.15.46:0/3019599374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2881409d0 0x7fa28806efb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 --2- 172.21.15.46:0/3019599374 >> v2:172.21.15.121:3300/0 conn(0x7fa28806abe0 0x7fa2881618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 -- 172.21.15.46:0/3019599374 >> 172.21.15.46:0/3019599374 conn(0x7fa2880b1e50 msgr2=0x7fa2880b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 -- 172.21.15.46:0/3019599374 shutdown_connections 2023-12-16T21:14:18.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.288+0000 7fa28c89b640 1 -- 172.21.15.46:0/3019599374 wait complete. 2023-12-16T21:14:18.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.289+0000 7fa28c89b640 1 Processor -- start 2023-12-16T21:14:18.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.289+0000 7fa28c89b640 1 -- start start 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28c89b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa28806abe0 0x7fa28813c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28c89b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa288140310 0x7fa288157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28c89b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 0x7fa288154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28c89b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28813e190 con 0x7fa2881409d0 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28c89b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa288143b30 con 0x7fa288140310 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28c89b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa288155a70 con 0x7fa28806abe0 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa286d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 0x7fa288154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa286d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 0x7fa288154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53494/0 (socket says 172.21.15.46:53494) 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa286d9d640 1 -- 172.21.15.46:0/1244535702 learned_addr learned my addr 172.21.15.46:0/1244535702 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa28659c640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa28806abe0 0x7fa28813c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.290+0000 7fa27ffff640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa288140310 0x7fa288157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa286d9d640 1 -- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa28806abe0 msgr2=0x7fa28813c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa286d9d640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa28806abe0 0x7fa28813c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa286d9d640 1 -- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa288140310 msgr2=0x7fa288157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa286d9d640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa288140310 0x7fa288157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa286d9d640 1 -- 172.21.15.46:0/1244535702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280009a70 con 0x7fa2881409d0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa286d9d640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 0x7fa288154b60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa280009d90 tx=0x7fa280004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa280040650 con 0x7fa2881409d0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28002ce90 con 0x7fa2881409d0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2800409f0 con 0x7fa2881409d0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28813dad0 con 0x7fa2881409d0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.291+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa288155d50 con 0x7fa2881409d0 2023-12-16T21:14:18.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.292+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28813e2d0 con 0x7fa2881409d0 2023-12-16T21:14:18.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.293+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa280040b50 con 0x7fa2881409d0 2023-12-16T21:14:18.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.293+0000 7fa27effd640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa26806ae70 0x7fa26806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.293+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa2800abdc0 con 0x7fa2881409d0 2023-12-16T21:14:18.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.293+0000 7fa28659c640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa26806ae70 0x7fa26806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.294+0000 7fa28659c640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa26806ae70 0x7fa26806d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa2780011e0 tx=0x7fa278007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.297+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa28813e2d0 con 0x7fa2881409d0 2023-12-16T21:14:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.471+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa28813e2d0 con 0x7fa2881409d0 2023-12-16T21:14:18.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.472+0000 7fa27effd640 1 -- 172.21.15.46:0/1244535702 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa28813e2d0 con 0x7fa2881409d0 2023-12-16T21:14:18.472 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa26806ae70 msgr2=0x7fa26806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa26806ae70 0x7fa26806d330 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa2780011e0 tx=0x7fa278007d30 comp rx=0 tx=0).stop 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 msgr2=0x7fa288154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 0x7fa288154b60 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa280009d90 tx=0x7fa280004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 shutdown_connections 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa26806ae70 0x7fa26806d330 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2881409d0 0x7fa288154b60 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa288140310 0x7fa288157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 --2- 172.21.15.46:0/1244535702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa28806abe0 0x7fa28813c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 >> 172.21.15.46:0/1244535702 conn(0x7fa2880b1e50 msgr2=0x7fa288140de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:18.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 shutdown_connections 2023-12-16T21:14:18.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.474+0000 7fa28c89b640 1 -- 172.21.15.46:0/1244535702 wait complete. 2023-12-16T21:14:18.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:18.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.594+0000 7f4474fc9640 1 Processor -- start 2023-12-16T21:14:18.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.594+0000 7f4474fc9640 1 -- start start 2023-12-16T21:14:18.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4474fc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4470143490 0x7f44700b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.596 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4474fc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015abf0 0x7f44700b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4474fc9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f447015e2a0 con 0x7f447015e730 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4474fc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447015b520 con 0x7f4470143490 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4474fc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4470143870 con 0x7f447015abf0 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446e59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4470143490 0x7f44700b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4467fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f447015e730 0x7f447015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33564/0 (socket says 172.21.15.46:33564) 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f4467fff640 1 -- 172.21.15.46:0/3232280893 learned_addr learned my addr 172.21.15.46:0/3232280893 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446ed9d640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015abf0 0x7f44700b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446e59c640 1 -- 172.21.15.46:0/3232280893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015abf0 msgr2=0x7f44700b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446e59c640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015abf0 0x7f44700b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446e59c640 1 -- 172.21.15.46:0/3232280893 >> v1:172.21.15.46:6789/0 conn(0x7f447015e730 legacy=0x7f447015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446e59c640 1 -- 172.21.15.46:0/3232280893 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447015b240 con 0x7f4470143490 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446dd9b640 1 -- 172.21.15.46:0/3232280893 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781328595 0 0) 0x7f447015e2a0 con 0x7f447015e730 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.595+0000 7f446ed9d640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015abf0 0x7f44700b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.596+0000 7f446e59c640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44700b0cf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f446000ba40 tx=0x7f446002e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0dc44073690eb4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.596+0000 7f446dd9b640 1 -- 172.21.15.46:0/3232280893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446002ee30 con 0x7f4470143490 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.596+0000 7f446dd9b640 1 -- 172.21.15.46:0/3232280893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460036440 con 0x7f4470143490 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.596+0000 7f446dd9b640 1 -- 172.21.15.46:0/3232280893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44600367d0 con 0x7f4470143490 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 -- 172.21.15.46:0/3232280893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 msgr2=0x7f44700b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44700b0cf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f446000ba40 tx=0x7f446002e080 comp rx=0 tx=0).stop 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 -- 172.21.15.46:0/3232280893 shutdown_connections 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015abf0 0x7f44700b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 --2- 172.21.15.46:0/3232280893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44700b0cf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 -- 172.21.15.46:0/3232280893 >> 172.21.15.46:0/3232280893 conn(0x7f447006a4d0 msgr2=0x7f447006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:18.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 -- 172.21.15.46:0/3232280893 shutdown_connections 2023-12-16T21:14:18.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.597+0000 7f4474fc9640 1 -- 172.21.15.46:0/3232280893 wait complete. 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.598+0000 7f4474fc9640 1 Processor -- start 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.598+0000 7f4474fc9640 1 -- start start 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f4474fc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44701e38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f4474fc9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447015abf0 0x7f44700ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f4474fc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015e730 0x7f44700ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f4474fc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700cb290 con 0x7f4470143490 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f4474fc9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44700c6240 con 0x7f447015abf0 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f4474fc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44700c5960 con 0x7f447015e730 2023-12-16T21:14:18.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44701e38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44701e38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53518/0 (socket says 172.21.15.46:53518) 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 -- 172.21.15.46:0/2820287603 learned_addr learned my addr 172.21.15.46:0/2820287603 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 -- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015e730 msgr2=0x7f44700ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015e730 0x7f44700ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 -- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447015abf0 msgr2=0x7f44700ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447015abf0 0x7f44700ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 -- 172.21.15.46:0/2820287603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446000b6a0 con 0x7f4470143490 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.599+0000 7f446e59c640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44701e38d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4460000c00 tx=0x7f4460004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.600+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460004230 con 0x7f4470143490 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.600+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460004390 con 0x7f4470143490 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.600+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460034b20 con 0x7f4470143490 2023-12-16T21:14:18.602 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.600+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44700cacd0 con 0x7f4470143490 2023-12-16T21:14:18.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.600+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44700c5e60 con 0x7f4470143490 2023-12-16T21:14:18.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.601+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44700c5cc0 con 0x7f4470143490 2023-12-16T21:14:18.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.601+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f446003d040 con 0x7f4470143490 2023-12-16T21:14:18.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.602+0000 7f4466ffd640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f445006ae40 0x7f445006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.602+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f44600ac600 con 0x7f4470143490 2023-12-16T21:14:18.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.602+0000 7f4467fff640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f445006ae40 0x7f445006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.603+0000 7f4467fff640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f445006ae40 0x7f445006d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4458007fd0 tx=0x7f4458007960 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.606+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f44700c5cc0 con 0x7f4470143490 2023-12-16T21:14:18.781 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.782+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44700c5cc0 con 0x7f4470143490 2023-12-16T21:14:18.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.782+0000 7f4466ffd640 1 -- 172.21.15.46:0/2820287603 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44700c5cc0 con 0x7f4470143490 2023-12-16T21:14:18.782 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:18.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f445006ae40 msgr2=0x7f445006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f445006ae40 0x7f445006d300 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4458007fd0 tx=0x7f4458007960 comp rx=0 tx=0).stop 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 msgr2=0x7f44701e38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44701e38d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4460000c00 tx=0x7f4460004730 comp rx=0 tx=0).stop 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 shutdown_connections 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f445006ae40 0x7f445006d300 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f447015e730 0x7f44700ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f447015abf0 0x7f44700ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 --2- 172.21.15.46:0/2820287603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4470143490 0x7f44701e38d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 >> 172.21.15.46:0/2820287603 conn(0x7f447006a4d0 msgr2=0x7f44700b3bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 shutdown_connections 2023-12-16T21:14:18.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.784+0000 7f4474fc9640 1 -- 172.21.15.46:0/2820287603 wait complete. 2023-12-16T21:14:18.794 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:18.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.904+0000 7f96c73a9640 1 Processor -- start 2023-12-16T21:14:18.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.905+0000 7f96c73a9640 1 -- start start 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.905+0000 7f96c73a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 0x7f96c00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.905+0000 7f96c73a9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c015e2a0 con 0x7f96c015abf0 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.905+0000 7f96c73a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c015b240 con 0x7f96c015e730 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.905+0000 7f96c73a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c015b520 con 0x7f96c0143490 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96c5946640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96c015abf0 0x7f96c00b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33572/0 (socket says 172.21.15.46:33572) 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96c5946640 1 -- 172.21.15.46:0/4112233028 learned_addr learned my addr 172.21.15.46:0/4112233028 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:18.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96c5145640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 0x7f96c00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396102501 0 0) 0x7f96c015e2a0 con 0x7f96c015abf0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ac003520 con 0x7f96c015abf0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238214379 0 0) 0x7f96ac003520 con 0x7f96c015abf0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 msgr2=0x7f96c00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 0x7f96c00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 >> v1:172.21.15.121:6789/0 conn(0x7f96c015e730 legacy=0x7f96c015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c015dc50 con 0x7f96c015abf0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f96b80019f0 con 0x7f96c015abf0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 >> v1:172.21.15.46:6789/0 conn(0x7f96c015abf0 legacy=0x7f96c00b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96ac0039a0 0x7f96ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 0x7f96ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96ac004d00 0x7f96ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c015b520 con 0x7f96ac0039a0 2023-12-16T21:14:18.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac00ba60 con 0x7f96ac004d00 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac00de30 con 0x7f96c0143490 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.906+0000 7f96c5145640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96ac0039a0 0x7f96ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5946640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96ac004d00 0x7f96ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c4944640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 0x7f96ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5145640 1 -- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 msgr2=0x7f96ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5145640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c0143490 0x7f96ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5145640 1 -- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96ac004d00 msgr2=0x7f96ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5145640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96ac004d00 0x7f96ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5145640 1 -- 172.21.15.46:0/4112233028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ac003520 con 0x7f96ac0039a0 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5946640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96ac004d00 0x7f96ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.907+0000 7f96c5145640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96ac0039a0 0x7f96ac003dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009a90 tx=0x7f96ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6b071d1b959cd6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc02c680 con 0x7f96ac0039a0 2023-12-16T21:14:18.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc02c7e0 con 0x7f96ac0039a0 2023-12-16T21:14:18.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96b7fff640 1 -- 172.21.15.46:0/4112233028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc02cb70 con 0x7f96ac0039a0 2023-12-16T21:14:18.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c73a9640 1 -- 172.21.15.46:0/4112233028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96ac0039a0 msgr2=0x7f96ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c73a9640 1 --2- 172.21.15.46:0/4112233028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96ac0039a0 0x7f96ac003dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009a90 tx=0x7f96ac0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c5946640 1 -- 172.21.15.46:0/4112233028 reap_dead start 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c73a9640 1 -- 172.21.15.46:0/4112233028 shutdown_connections 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c73a9640 1 -- 172.21.15.46:0/4112233028 >> 172.21.15.46:0/4112233028 conn(0x7f96c006a4d0 msgr2=0x7f96c006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c73a9640 1 -- 172.21.15.46:0/4112233028 shutdown_connections 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.908+0000 7f96c73a9640 1 -- 172.21.15.46:0/4112233028 wait complete. 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.909+0000 7f96c73a9640 1 Processor -- start 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.909+0000 7f96c73a9640 1 -- start start 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c73a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96ac0039a0 0x7f96c01598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c73a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c00ae360 0x7f96c013f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c73a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 0x7f96c00aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c73a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0140f10 con 0x7f96c013f9d0 2023-12-16T21:14:18.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c73a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c013fe70 con 0x7f96c00ae360 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c73a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c0140150 con 0x7f96ac0039a0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 0x7f96c00aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 0x7f96c00aaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53542/0 (socket says 172.21.15.46:53542) 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 -- 172.21.15.46:0/38986538 learned_addr learned my addr 172.21.15.46:0/38986538 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c4944640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c00ae360 0x7f96c013f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5145640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96ac0039a0 0x7f96c01598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 -- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96ac0039a0 msgr2=0x7f96c01598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96ac0039a0 0x7f96c01598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 -- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c00ae360 msgr2=0x7f96c013f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c00ae360 0x7f96c013f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.910+0000 7f96c5946640 1 -- 172.21.15.46:0/38986538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c0140430 con 0x7f96c013f9d0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.911+0000 7f96c5946640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 0x7f96c00aaeb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f96b8001370 tx=0x7f96b800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.911+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8007ae0 con 0x7f96c013f9d0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.911+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b800eda0 con 0x7f96c013f9d0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.911+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c0140710 con 0x7f96c013f9d0 2023-12-16T21:14:18.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.911+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8013640 con 0x7f96c013f9d0 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.911+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c0159e10 con 0x7f96c013f9d0 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.912+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c0140c30 con 0x7f96c013f9d0 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.912+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f96b80137a0 con 0x7f96c013f9d0 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.913+0000 7f96b5ffb640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96a406ae70 0x7f96a406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.913+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f96b8089b10 con 0x7f96c013f9d0 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.913+0000 7f96c5145640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96a406ae70 0x7f96a406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:18.915 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.914+0000 7f96c5145640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96a406ae70 0x7f96a406d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f96bc036000 tx=0x7f96bc030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:18.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:18.917+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f96c0140c30 con 0x7f96c013f9d0 2023-12-16T21:14:19.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.094+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f96c0140c30 con 0x7f96c013f9d0 2023-12-16T21:14:19.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.094+0000 7f96b5ffb640 1 -- 172.21.15.46:0/38986538 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f96c0140c30 con 0x7f96c013f9d0 2023-12-16T21:14:19.094 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96a406ae70 msgr2=0x7f96a406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96a406ae70 0x7f96a406d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f96bc036000 tx=0x7f96bc030000 comp rx=0 tx=0).stop 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 msgr2=0x7f96c00aaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 0x7f96c00aaeb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f96b8001370 tx=0x7f96b800e530 comp rx=0 tx=0).stop 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 shutdown_connections 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96a406ae70 0x7f96a406d330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c013f9d0 0x7f96c00aaeb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c00ae360 0x7f96c013f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 --2- 172.21.15.46:0/38986538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96ac0039a0 0x7f96c01598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 >> 172.21.15.46:0/38986538 conn(0x7f96c006a4d0 msgr2=0x7f96c01418e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 shutdown_connections 2023-12-16T21:14:19.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.096+0000 7f96c73a9640 1 -- 172.21.15.46:0/38986538 wait complete. 2023-12-16T21:14:19.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:19.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.215+0000 7ff419b01640 1 Processor -- start 2023-12-16T21:14:19.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.215+0000 7ff419b01640 1 -- start start 2023-12-16T21:14:19.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff419b01640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff4140ceb70 0x7ff4140cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff419b01640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff4140b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff419b01640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff414071770 con 0x7ff414144980 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff419b01640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41415ced0 con 0x7ff4140ceb70 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff419b01640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4140b4ce0 con 0x7ff4140b4370 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff4137fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff414144980 0x7ff41415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:33574/0 (socket says 172.21.15.46:33574) 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff4137fe640 1 -- 172.21.15.46:0/1717795640 learned_addr learned my addr 172.21.15.46:0/1717795640 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff412ffd640 1 --2- 172.21.15.46:0/1717795640 >> v2:172.21.15.121:3300/0 conn(0x7ff4140ceb70 0x7ff4140cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff413fff640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff4140b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436115912 0 0) 0x7ff414071770 con 0x7ff414144980 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.216+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3fc003520 con 0x7ff414144980 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2523923114 0 0) 0x7ff3fc003520 con 0x7ff414144980 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 msgr2=0x7ff4140b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff4140b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 >> v2:172.21.15.121:3300/0 conn(0x7ff4140ceb70 msgr2=0x7ff4140cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 --2- 172.21.15.46:0/1717795640 >> v2:172.21.15.121:3300/0 conn(0x7ff4140ceb70 0x7ff4140cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4140ccdc0 con 0x7ff414144980 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7ff404001670 con 0x7ff414144980 2023-12-16T21:14:19.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff413fff640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff4140b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff412ffd640 1 --2- 172.21.15.46:0/1717795640 >> v2:172.21.15.121:3300/0 conn(0x7ff4140ceb70 0x7ff4140cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 >> v1:172.21.15.46:6789/0 conn(0x7ff414144980 legacy=0x7ff41415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff3fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc004d00 0x7ff3fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41415ced0 con 0x7ff3fc003970 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4140b4ce0 con 0x7ff3fc004d00 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3fc00b980 con 0x7ff4140b4370 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff413fff640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc004d00 0x7ff3fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff412ffd640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff3fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4137fe640 1 -- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 msgr2=0x7ff3fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff4140b4370 0x7ff3fc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4137fe640 1 -- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc004d00 msgr2=0x7ff3fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:19.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc004d00 0x7ff3fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.217+0000 7ff4137fe640 1 -- 172.21.15.46:0/1717795640 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c009a70 con 0x7ff3fc003970 2023-12-16T21:14:19.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c240 tx=0x7ff3fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8d8f669b4e9fbf server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:19.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40402cb00 con 0x7ff3fc003970 2023-12-16T21:14:19.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40402cc60 con 0x7ff3fc003970 2023-12-16T21:14:19.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff4127fc640 1 -- 172.21.15.46:0/1717795640 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404005ad0 con 0x7ff3fc003970 2023-12-16T21:14:19.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff419b01640 1 -- 172.21.15.46:0/1717795640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3fc003970 msgr2=0x7ff3fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff419b01640 1 --2- 172.21.15.46:0/1717795640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3fc003970 0x7ff3fc003d90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff40400c240 tx=0x7ff3fc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.218+0000 7ff413fff640 1 -- 172.21.15.46:0/1717795640 reap_dead start 2023-12-16T21:14:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.219+0000 7ff419b01640 1 -- 172.21.15.46:0/1717795640 shutdown_connections 2023-12-16T21:14:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.219+0000 7ff419b01640 1 -- 172.21.15.46:0/1717795640 >> 172.21.15.46:0/1717795640 conn(0x7ff41406a4d0 msgr2=0x7ff41406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:19.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.219+0000 7ff419b01640 1 -- 172.21.15.46:0/1717795640 shutdown_connections 2023-12-16T21:14:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.219+0000 7ff419b01640 1 -- 172.21.15.46:0/1717795640 wait complete. 2023-12-16T21:14:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.220+0000 7ff419b01640 1 Processor -- start 2023-12-16T21:14:19.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.220+0000 7ff419b01640 1 -- start start 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff419b01640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc003970 0x7ff4140b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff419b01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 0x7ff414158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff419b01640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff414135940 0x7ff414135dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff419b01640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff414158650 con 0x7ff414157cb0 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff419b01640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff414133820 con 0x7ff3fc003970 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff419b01640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4141366e0 con 0x7ff414135940 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 0x7ff414158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 0x7ff414158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53562/0 (socket says 172.21.15.46:53562) 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 -- 172.21.15.46:0/1633391832 learned_addr learned my addr 172.21.15.46:0/1633391832 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc003970 0x7ff4140b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff413fff640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff414135940 0x7ff414135dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 -- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff414135940 msgr2=0x7ff414135dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff414135940 0x7ff414135dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 -- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc003970 msgr2=0x7ff4140b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc003970 0x7ff4140b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 -- 172.21.15.46:0/1633391832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff414133ae0 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff412ffd640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 0x7ff414158110 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff408011cd0 tx=0x7ff40800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.221+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800fd20 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.222+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40800fe80 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.222+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff414133dc0 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.222+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408004270 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.222+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4140c6d50 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.222+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff414069400 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.223+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff4080043d0 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.223+0000 7ff3fbfff640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff40006ae40 0x7ff40006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.223+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff408089d00 con 0x7ff414157cb0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.223+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff40006ae40 0x7ff40006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.224+0000 7ff4137fe640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff40006ae40 0x7ff40006d300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff404034000 tx=0x7ff404007000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:19.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.228+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff408054d50 con 0x7ff414157cb0 2023-12-16T21:14:19.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.415+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff414159210 con 0x7ff414157cb0 2023-12-16T21:14:19.415 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.416+0000 7ff3fbfff640 1 -- 172.21.15.46:0/1633391832 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff408022080 con 0x7ff414157cb0 2023-12-16T21:14:19.415 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:19.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff40006ae40 msgr2=0x7ff40006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff40006ae40 0x7ff40006d300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff404034000 tx=0x7ff404007000 comp rx=0 tx=0).stop 2023-12-16T21:14:19.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 msgr2=0x7ff414158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 0x7ff414158110 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff408011cd0 tx=0x7ff40800f610 comp rx=0 tx=0).stop 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 shutdown_connections 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff40006ae40 0x7ff40006d300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff414157cb0 0x7ff414158110 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff414135940 0x7ff414135dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 --2- 172.21.15.46:0/1633391832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3fc003970 0x7ff4140b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 >> 172.21.15.46:0/1633391832 conn(0x7ff41406a4d0 msgr2=0x7ff4141553b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 shutdown_connections 2023-12-16T21:14:19.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.418+0000 7ff419b01640 1 -- 172.21.15.46:0/1633391832 wait complete. 2023-12-16T21:14:19.428 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:14:19.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:19.429 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-12-16T21:14:19.460 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:19.460 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:14:19.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:19.460 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-16T21:14:19.516 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:14:19.516 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:19.535 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:14:19.535 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:19.594 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.702+0000 7f891964f640 1 Processor -- start 2023-12-16T21:14:19.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.702+0000 7f891964f640 1 -- start start 2023-12-16T21:14:19.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f891964f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89140b1400 0x7f891415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f891964f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f891415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f891964f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891415e2a0 con 0x7f8914143490 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f891964f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891415b520 con 0x7f89140b1400 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f891964f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8914143870 con 0x7f891415e730 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f89137fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f891415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f8912ffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8914143490 0x7f89140b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57914/0 (socket says 172.21.15.46:57914) 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f89127fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89140b1400 0x7f891415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.703+0000 7f8912ffd640 1 -- 172.21.15.46:0/2177447783 learned_addr learned my addr 172.21.15.46:0/2177447783 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f8911ffb640 1 -- 172.21.15.46:0/2177447783 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044875673 0 0) 0x7f891415e2a0 con 0x7f8914143490 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f8911ffb640 1 -- 172.21.15.46:0/2177447783 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8900003520 con 0x7f8914143490 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f89127fc640 1 -- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 msgr2=0x7f891415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f89127fc640 1 --2- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f891415aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f89127fc640 1 -- 172.21.15.46:0/2177447783 >> v1:172.21.15.121:6789/0 conn(0x7f8914143490 legacy=0x7f89140b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f89127fc640 1 -- 172.21.15.46:0/2177447783 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891415b240 con 0x7f89140b1400 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.704+0000 7f89137fe640 1 --2- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f891415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f89127fc640 1 --2- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f891415a4e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0097a0 tx=0x7f88fc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467ac6b8332ee814 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f8911ffb640 1 -- 172.21.15.46:0/2177447783 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc02d040 con 0x7f89140b1400 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f8911ffb640 1 -- 172.21.15.46:0/2177447783 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc031810 con 0x7f89140b1400 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f8911ffb640 1 -- 172.21.15.46:0/2177447783 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc031ba0 con 0x7f89140b1400 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 -- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 msgr2=0x7f891415a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 --2- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f891415a4e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f88fc0097a0 tx=0x7f88fc02b8b0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 -- 172.21.15.46:0/2177447783 shutdown_connections 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 --2- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f891415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 --2- 172.21.15.46:0/2177447783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f891415a4e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 -- 172.21.15.46:0/2177447783 >> 172.21.15.46:0/2177447783 conn(0x7f891406a4d0 msgr2=0x7f891406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 -- 172.21.15.46:0/2177447783 shutdown_connections 2023-12-16T21:14:19.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.705+0000 7f891964f640 1 -- 172.21.15.46:0/2177447783 wait complete. 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.707+0000 7f891964f640 1 Processor -- start 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.707+0000 7f891964f640 1 -- start start 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.707+0000 7f891964f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f89140ad420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f891964f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8914143490 0x7f8914158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f891964f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f89140cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f891964f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89140ad000 con 0x7f8914143490 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f891964f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89140ad960 con 0x7f89140b1400 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f891964f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89140ac980 con 0x7f891415e730 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f89140ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f89127fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8914143490 0x7f8914158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f89140ad420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:55892/0 (socket says 172.21.15.46:55892) 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 -- 172.21.15.46:0/745903834 learned_addr learned my addr 172.21.15.46:0/745903834 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f89137fe640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f89140cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 -- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 msgr2=0x7f89140cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f89140cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 -- 172.21.15.46:0/745903834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8914143490 msgr2=0x7f8914158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8914143490 0x7f8914158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 -- 172.21.15.46:0/745903834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890400bc80 con 0x7f89140b1400 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f89127fc640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8914143490 0x7f8914158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.708+0000 7f8912ffd640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f89140ad420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f890c00c500 tx=0x7f890c00c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.709+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c005ce0 con 0x7f89140b1400 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.709+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890c005e40 con 0x7f89140b1400 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.709+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890c007330 con 0x7f89140b1400 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.709+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc009400 con 0x7f89140b1400 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.709+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89140acc20 con 0x7f89140b1400 2023-12-16T21:14:19.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.710+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8914069400 con 0x7f89140b1400 2023-12-16T21:14:19.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.710+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f890c007490 con 0x7f89140b1400 2023-12-16T21:14:19.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.710+0000 7f890b7fe640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88f406ae70 0x7f88f406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:19.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.711+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f890c088bd0 con 0x7f89140b1400 2023-12-16T21:14:19.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.711+0000 7f89127fc640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88f406ae70 0x7f88f406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:19.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.711+0000 7f89127fc640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88f406ae70 0x7f88f406d330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f88fc036000 tx=0x7f88fc03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:19.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.716+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f890c053e00 con 0x7f89140b1400 2023-12-16T21:14:19.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.892+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89140b2820 con 0x7f89140b1400 2023-12-16T21:14:19.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.893+0000 7f890b7fe640 1 -- 172.21.15.46:0/745903834 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f890c017020 con 0x7f89140b1400 2023-12-16T21:14:19.893 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:19.893 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:17.654454+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88f406ae70 msgr2=0x7f88f406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88f406ae70 0x7f88f406d330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f88fc036000 tx=0x7f88fc03c040 comp rx=0 tx=0).stop 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 msgr2=0x7f89140ad420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f89140ad420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f890c00c500 tx=0x7f890c00c9c0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 shutdown_connections 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88f406ae70 0x7f88f406d330 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8914143490 0x7f8914158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f891415e730 0x7f89140cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 --2- 172.21.15.46:0/745903834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89140b1400 0x7f89140ad420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:19.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 >> 172.21.15.46:0/745903834 conn(0x7f891406a4d0 msgr2=0x7f891406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:19.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 shutdown_connections 2023-12-16T21:14:19.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:19.895+0000 7f891964f640 1 -- 172.21.15.46:0/745903834 wait complete. 2023-12-16T21:14:19.906 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:19.906 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:20.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.017+0000 7f859fa26640 1 Processor -- start 2023-12-16T21:14:20.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.017+0000 7f859fa26640 1 -- start start 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859fa26640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85980ad640 0x7f859806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859fa26640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f859806bbf0 0x7f859806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859fa26640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980adae0 con 0x7f859815d1d0 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859fa26640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85980addc0 con 0x7f859806bbf0 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859fa26640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85980ae0a0 con 0x7f85980ad640 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f859806bbf0 0x7f859806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f859806bbf0 0x7f859806d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53580/0 (socket says 172.21.15.46:53580) 2023-12-16T21:14:20.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 -- 172.21.15.46:0/760427929 learned_addr learned my addr 172.21.15.46:0/760427929 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:20.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859cfc1640 1 --2- 172.21.15.46:0/760427929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85980ad640 0x7f859806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 -- 172.21.15.46:0/760427929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85980ad640 msgr2=0x7f859806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:20.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 --2- 172.21.15.46:0/760427929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85980ad640 0x7f859806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 -- 172.21.15.46:0/760427929 >> v1:172.21.15.121:6789/0 conn(0x7f859815d1d0 legacy=0x7f859806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.018+0000 7f859dfc3640 1 -- 172.21.15.46:0/760427929 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85980ae380 con 0x7f859806bbf0 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.019+0000 7f859dfc3640 1 --2- 172.21.15.46:0/760427929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f859806d5f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8590009e80 tx=0x7f859002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af64ef4559a4d7c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.019+0000 7f858ffff640 1 -- 172.21.15.46:0/760427929 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859002caf0 con 0x7f859806bbf0 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.019+0000 7f858ffff640 1 -- 172.21.15.46:0/760427929 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859002cc50 con 0x7f859806bbf0 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.019+0000 7f858ffff640 1 -- 172.21.15.46:0/760427929 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900343f0 con 0x7f859806bbf0 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 -- 172.21.15.46:0/760427929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 msgr2=0x7f859806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 --2- 172.21.15.46:0/760427929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f859806d5f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8590009e80 tx=0x7f859002bf50 comp rx=0 tx=0).stop 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 -- 172.21.15.46:0/760427929 shutdown_connections 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 --2- 172.21.15.46:0/760427929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85980ad640 0x7f859806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 --2- 172.21.15.46:0/760427929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f859806d5f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 -- 172.21.15.46:0/760427929 >> 172.21.15.46:0/760427929 conn(0x7f8598070e70 msgr2=0x7f8598071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 -- 172.21.15.46:0/760427929 shutdown_connections 2023-12-16T21:14:20.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.020+0000 7f859fa26640 1 -- 172.21.15.46:0/760427929 wait complete. 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.021+0000 7f859fa26640 1 Processor -- start 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.021+0000 7f859fa26640 1 -- start start 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859fa26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f85980ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859fa26640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85980ad640 0x7f859815ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859fa26640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f859815d1d0 0x7f85981f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859fa26640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859815fe40 con 0x7f859806bbf0 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859fa26640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859815d9e0 con 0x7f85980ad640 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859fa26640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859815dcc0 con 0x7f859815d1d0 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f85980ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f85980ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53582/0 (socket says 172.21.15.46:53582) 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 -- 172.21.15.46:0/1641783286 learned_addr learned my addr 172.21.15.46:0/1641783286 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:20.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859cfc1640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85980ad640 0x7f859815ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859dfc3640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f859815d1d0 0x7f85981f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 -- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f859815d1d0 msgr2=0x7f85981f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f859815d1d0 0x7f85981f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 -- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85980ad640 msgr2=0x7f859815ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85980ad640 0x7f859815ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 -- 172.21.15.46:0/1641783286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8590009b30 con 0x7f859806bbf0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.022+0000 7f859d7c2640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f85980ce8d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f859400c3c0 tx=0x7f859400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.023+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940052d0 con 0x7f859806bbf0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.023+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594005450 con 0x7f859806bbf0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.023+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859815dfa0 con 0x7f859806bbf0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.023+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594007260 con 0x7f859806bbf0 2023-12-16T21:14:20.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.023+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859815e560 con 0x7f859806bbf0 2023-12-16T21:14:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.024+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598069400 con 0x7f859806bbf0 2023-12-16T21:14:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.024+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f85940073c0 con 0x7f859806bbf0 2023-12-16T21:14:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.025+0000 7f858dffb640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f857c06ae40 0x7f857c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.025+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8594088900 con 0x7f859806bbf0 2023-12-16T21:14:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.025+0000 7f859cfc1640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f857c06ae40 0x7f857c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.025+0000 7f859cfc1640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f857c06ae40 0x7f857c06d300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8598160c70 tx=0x7f8588009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.029+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8594053a70 con 0x7f859806bbf0 2023-12-16T21:14:20.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.204+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8598071380 con 0x7f859806bbf0 2023-12-16T21:14:20.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.204+0000 7f858dffb640 1 -- 172.21.15.46:0/1641783286 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8594057090 con 0x7f859806bbf0 2023-12-16T21:14:20.204 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f857c06ae40 msgr2=0x7f857c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f857c06ae40 0x7f857c06d300 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8598160c70 tx=0x7f8588009190 comp rx=0 tx=0).stop 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 msgr2=0x7f85980ce8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f85980ce8d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f859400c3c0 tx=0x7f859400c880 comp rx=0 tx=0).stop 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 shutdown_connections 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f857c06ae40 0x7f857c06d300 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f859815d1d0 0x7f85981f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85980ad640 0x7f859815ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 --2- 172.21.15.46:0/1641783286 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f859806bbf0 0x7f85980ce8d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 >> 172.21.15.46:0/1641783286 conn(0x7f8598070e70 msgr2=0x7f859806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 shutdown_connections 2023-12-16T21:14:20.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.206+0000 7f859fa26640 1 -- 172.21.15.46:0/1641783286 wait complete. 2023-12-16T21:14:20.216 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 Processor -- start 2023-12-16T21:14:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 -- start start 2023-12-16T21:14:20.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff3c40d2910 0x7ff3c40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff3c40d3730 0x7ff3c40b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c40b2b10 con 0x7ff3c415d330 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c40d3bc0 con 0x7ff3c40d3730 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.327+0000 7ff3cb168640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c40d1590 con 0x7ff3c40d2910 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c3fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff3c40d2910 0x7ff3c40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c8f04640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff3c415d330 0x7ff3c40b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57928/0 (socket says 172.21.15.46:57928) 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c8f04640 1 -- 172.21.15.46:0/1622953284 learned_addr learned my addr 172.21.15.46:0/1622953284 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c9705640 1 --2- 172.21.15.46:0/1622953284 >> v2:172.21.15.46:3300/0 conn(0x7ff3c40d3730 0x7ff3c40b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c3fff640 1 -- 172.21.15.46:0/1622953284 >> v2:172.21.15.46:3300/0 conn(0x7ff3c40d3730 msgr2=0x7ff3c40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c37fe640 1 -- 172.21.15.46:0/1622953284 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519956317 0 0) 0x7ff3c40b2b10 con 0x7ff3c415d330 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c3fff640 1 --2- 172.21.15.46:0/1622953284 >> v2:172.21.15.46:3300/0 conn(0x7ff3c40d3730 0x7ff3c40b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c3fff640 1 -- 172.21.15.46:0/1622953284 >> v1:172.21.15.121:6789/0 conn(0x7ff3c415d330 legacy=0x7ff3c40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c3fff640 1 -- 172.21.15.46:0/1622953284 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c40d1870 con 0x7ff3c40d2910 2023-12-16T21:14:20.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.328+0000 7ff3c9705640 1 --2- 172.21.15.46:0/1622953284 >> v2:172.21.15.46:3300/0 conn(0x7ff3c40d3730 0x7ff3c40b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.329+0000 7ff3c3fff640 1 --2- 172.21.15.46:0/1622953284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c40d2910 0x7ff3c40d31f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40d1540 tx=0x7ff3b002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=136840b1753ddd5e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.329+0000 7ff3c37fe640 1 -- 172.21.15.46:0/1622953284 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b002e070 con 0x7ff3c40d2910 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.329+0000 7ff3c37fe640 1 -- 172.21.15.46:0/1622953284 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b00315a0 con 0x7ff3c40d2910 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.329+0000 7ff3c37fe640 1 -- 172.21.15.46:0/1622953284 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0031930 con 0x7ff3c40d2910 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.329+0000 7ff3cb168640 1 -- 172.21.15.46:0/1622953284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c40d2910 msgr2=0x7ff3c40d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.329+0000 7ff3cb168640 1 --2- 172.21.15.46:0/1622953284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c40d2910 0x7ff3c40d31f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3c40d1540 tx=0x7ff3b002b840 comp rx=0 tx=0).stop 2023-12-16T21:14:20.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.330+0000 7ff3cb168640 1 -- 172.21.15.46:0/1622953284 shutdown_connections 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.330+0000 7ff3cb168640 1 --2- 172.21.15.46:0/1622953284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c40d2910 0x7ff3c40d31f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.330+0000 7ff3cb168640 1 --2- 172.21.15.46:0/1622953284 >> v2:172.21.15.46:3300/0 conn(0x7ff3c40d3730 0x7ff3c40b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.330+0000 7ff3cb168640 1 -- 172.21.15.46:0/1622953284 >> 172.21.15.46:0/1622953284 conn(0x7ff3c4072020 msgr2=0x7ff3c4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.330+0000 7ff3cb168640 1 -- 172.21.15.46:0/1622953284 shutdown_connections 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.330+0000 7ff3cb168640 1 -- 172.21.15.46:0/1622953284 wait complete. 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.331+0000 7ff3cb168640 1 Processor -- start 2023-12-16T21:14:20.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.331+0000 7ff3cb168640 1 -- start start 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3cb168640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3c40d2910 0x7ff3c40c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3cb168640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 0x7ff3c415f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3cb168640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c415d330 0x7ff3c40b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3cb168640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4160920 con 0x7ff3c40d3730 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3cb168640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c415fda0 con 0x7ff3c415d330 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3cb168640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4160080 con 0x7ff3c40d2910 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 0x7ff3c415f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 0x7ff3c415f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53604/0 (socket says 172.21.15.46:53604) 2023-12-16T21:14:20.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c8f04640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3c40d2910 0x7ff3c40c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 -- 172.21.15.46:0/3380523957 learned_addr learned my addr 172.21.15.46:0/3380523957 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c9705640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c415d330 0x7ff3c40b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 -- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3c40d2910 msgr2=0x7ff3c40c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3c40d2910 0x7ff3c40c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 -- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c415d330 msgr2=0x7ff3c40b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c415d330 0x7ff3c40b3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 -- 172.21.15.46:0/3380523957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0009400 con 0x7ff3c40d3730 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c3fff640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 0x7ff3c415f3e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0034040 tx=0x7ff3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0004230 con 0x7ff3c40d3730 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.332+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b0004390 con 0x7ff3c40d3730 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.333+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c4160360 con 0x7ff3c40d3730 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.333+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0032880 con 0x7ff3c40d3730 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.333+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c415f920 con 0x7ff3c40d3730 2023-12-16T21:14:20.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.333+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c4069400 con 0x7ff3c40d3730 2023-12-16T21:14:20.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.334+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff3b00329e0 con 0x7ff3c40d3730 2023-12-16T21:14:20.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.334+0000 7ff3c17fa640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff3a806ae40 0x7ff3a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.334+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff3b00ab690 con 0x7ff3c40d3730 2023-12-16T21:14:20.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.335+0000 7ff3c8f04640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff3a806ae40 0x7ff3a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.335+0000 7ff3c8f04640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff3a806ae40 0x7ff3a806d300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007fd0 tx=0x7ff3b8007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.338+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff3b0076760 con 0x7ff3c40d3730 2023-12-16T21:14:20.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.513+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3c406be50 con 0x7ff3c40d3730 2023-12-16T21:14:20.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.513+0000 7ff3c17fa640 1 -- 172.21.15.46:0/3380523957 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3b0079d80 con 0x7ff3c40d3730 2023-12-16T21:14:20.513 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:20.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff3a806ae40 msgr2=0x7ff3a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff3a806ae40 0x7ff3a806d300 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8007fd0 tx=0x7ff3b8007de0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 msgr2=0x7ff3c415f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 0x7ff3c415f3e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0034040 tx=0x7ff3b00046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 shutdown_connections 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff3a806ae40 0x7ff3a806d300 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff3c40d3730 0x7ff3c415f3e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3c415d330 0x7ff3c40b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 --2- 172.21.15.46:0/3380523957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff3c40d2910 0x7ff3c40c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 >> 172.21.15.46:0/3380523957 conn(0x7ff3c4072020 msgr2=0x7ff3c40b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 shutdown_connections 2023-12-16T21:14:20.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.515+0000 7ff3cb168640 1 -- 172.21.15.46:0/3380523957 wait complete. 2023-12-16T21:14:20.526 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.636+0000 7ff581dfc640 1 Processor -- start 2023-12-16T21:14:20.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.636+0000 7ff581dfc640 1 -- start start 2023-12-16T21:14:20.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff581dfc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff57c146460 0x7ff57c0b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff581dfc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c146fd0 con 0x7ff57c146840 2023-12-16T21:14:20.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff581dfc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff57c06c180 con 0x7ff57c06f1c0 2023-12-16T21:14:20.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff581dfc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff57c06c460 con 0x7ff57c146460 2023-12-16T21:14:20.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff57affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff57c146840 0x7ff57c06eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44962/0 (socket says 172.21.15.46:44962) 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff57affd640 1 -- 172.21.15.46:0/1977608154 learned_addr learned my addr 172.21.15.46:0/1977608154 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff57b7fe640 1 --2- 172.21.15.46:0/1977608154 >> v2:172.21.15.121:3300/0 conn(0x7ff57c146460 0x7ff57c0b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff57a7fc640 1 -- 172.21.15.46:0/1977608154 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154706229 0 0) 0x7ff57c146fd0 con 0x7ff57c146840 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff57a7fc640 1 -- 172.21.15.46:0/1977608154 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff564003520 con 0x7ff57c146840 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.637+0000 7ff57b7fe640 1 -- 172.21.15.46:0/1977608154 >> v1:172.21.15.121:6789/0 conn(0x7ff57c06f1c0 legacy=0x7ff57c15db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.638+0000 7ff57b7fe640 1 -- 172.21.15.46:0/1977608154 >> v1:172.21.15.46:6789/0 conn(0x7ff57c146840 legacy=0x7ff57c06eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.638+0000 7ff57b7fe640 1 -- 172.21.15.46:0/1977608154 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c06c740 con 0x7ff57c146460 2023-12-16T21:14:20.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.638+0000 7ff57b7fe640 1 --2- 172.21.15.46:0/1977608154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c146460 0x7ff57c0b49d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0097a0 tx=0x7ff56c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad06dfa1886f07db server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.638+0000 7ff57a7fc640 1 -- 172.21.15.46:0/1977608154 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c030610 con 0x7ff57c146460 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff57a7fc640 1 -- 172.21.15.46:0/1977608154 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c030770 con 0x7ff57c146460 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff57a7fc640 1 -- 172.21.15.46:0/1977608154 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c030ae0 con 0x7ff57c146460 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 -- 172.21.15.46:0/1977608154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c146460 msgr2=0x7ff57c0b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 --2- 172.21.15.46:0/1977608154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c146460 0x7ff57c0b49d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff56c0097a0 tx=0x7ff56c02b960 comp rx=0 tx=0).stop 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 -- 172.21.15.46:0/1977608154 shutdown_connections 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 --2- 172.21.15.46:0/1977608154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c146460 0x7ff57c0b49d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 -- 172.21.15.46:0/1977608154 >> 172.21.15.46:0/1977608154 conn(0x7ff57c06a6b0 msgr2=0x7ff57c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 -- 172.21.15.46:0/1977608154 shutdown_connections 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.639+0000 7ff581dfc640 1 -- 172.21.15.46:0/1977608154 wait complete. 2023-12-16T21:14:20.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.640+0000 7ff581dfc640 1 Processor -- start 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.640+0000 7ff581dfc640 1 -- start start 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff581dfc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c06f1c0 0x7ff57c0cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff581dfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 0x7ff57c0d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff581dfc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff57c146840 0x7ff57c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff581dfc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0cda20 con 0x7ff57c146460 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff581dfc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c0d19f0 con 0x7ff57c06f1c0 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff581dfc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57c134400 con 0x7ff57c146840 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff57affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 0x7ff57c0d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff57affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 0x7ff57c0d14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51860/0 (socket says 172.21.15.46:51860) 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff57affd640 1 -- 172.21.15.46:0/3914093183 learned_addr learned my addr 172.21.15.46:0/3914093183 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff57b7fe640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c06f1c0 0x7ff57c0cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.641+0000 7ff57bfff640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff57c146840 0x7ff57c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57affd640 1 -- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff57c146840 msgr2=0x7ff57c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57affd640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff57c146840 0x7ff57c15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57affd640 1 -- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c06f1c0 msgr2=0x7ff57c0cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57affd640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c06f1c0 0x7ff57c0cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57affd640 1 -- 172.21.15.46:0/3914093183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c009400 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57bfff640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff57c146840 0x7ff57c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff57affd640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 0x7ff57c0d14b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff57000c390 tx=0x7ff57000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff570005ce0 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff570005e40 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff57c0cce10 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5700072d0 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.642+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff57c0cd3d0 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.643+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff57c069400 con 0x7ff57c146460 2023-12-16T21:14:20.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.643+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff5700074f0 con 0x7ff57c146460 2023-12-16T21:14:20.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.644+0000 7ff563fff640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff56806ae40 0x7ff56806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.644+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff570088b40 con 0x7ff57c146460 2023-12-16T21:14:20.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.644+0000 7ff57b7fe640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff56806ae40 0x7ff56806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.645+0000 7ff57b7fe640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff56806ae40 0x7ff56806d300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff56c036000 tx=0x7ff56c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.648+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff570053c30 con 0x7ff57c146460 2023-12-16T21:14:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.824+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff57c144140 con 0x7ff57c146460 2023-12-16T21:14:20.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.824+0000 7ff563fff640 1 -- 172.21.15.46:0/3914093183 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff570057250 con 0x7ff57c146460 2023-12-16T21:14:20.824 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff56806ae40 msgr2=0x7ff56806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff56806ae40 0x7ff56806d300 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff56c036000 tx=0x7ff56c008030 comp rx=0 tx=0).stop 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 msgr2=0x7ff57c0d14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 0x7ff57c0d14b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff57000c390 tx=0x7ff57000c850 comp rx=0 tx=0).stop 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 shutdown_connections 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff56806ae40 0x7ff56806d300 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff57c146460 0x7ff57c0d14b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff57c146840 0x7ff57c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 --2- 172.21.15.46:0/3914093183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff57c06f1c0 0x7ff57c0cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.826+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 >> 172.21.15.46:0/3914093183 conn(0x7ff57c06a6b0 msgr2=0x7ff57c06d070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.827+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 shutdown_connections 2023-12-16T21:14:20.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.827+0000 7ff581dfc640 1 -- 172.21.15.46:0/3914093183 wait complete. 2023-12-16T21:14:20.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:20.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.945+0000 7f53102c9640 1 Processor -- start 2023-12-16T21:14:20.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.945+0000 7f53102c9640 1 -- start start 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f53102c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5308142270 0x7f53081426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f53102c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5308161d20 con 0x7f5308160fb0 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f53102c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f530815e760 con 0x7f530806ea50 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f53102c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53080d1ee0 con 0x7f5308142270 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530e866640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5308142270 0x7f53081426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530e866640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5308142270 0x7f53081426c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51872/0 (socket says 172.21.15.46:51872) 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530d864640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5308160fb0 0x7f5308161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44964/0 (socket says 172.21.15.46:44964) 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530e866640 1 -- 172.21.15.46:0/2617236438 learned_addr learned my addr 172.21.15.46:0/2617236438 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:20.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530e866640 1 -- 172.21.15.46:0/2617236438 >> v1:172.21.15.121:6789/0 conn(0x7f530806ea50 legacy=0x7f530806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530e866640 1 -- 172.21.15.46:0/2617236438 >> v1:172.21.15.46:6789/0 conn(0x7f5308160fb0 legacy=0x7f5308161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.946+0000 7f530e866640 1 -- 172.21.15.46:0/2617236438 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530815d9a0 con 0x7f5308142270 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.947+0000 7f530e866640 1 --2- 172.21.15.46:0/2617236438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5308142270 0x7f53081426c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5300009e50 tx=0x7f530002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67efe8314f2be405 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.947+0000 7f530d063640 1 -- 172.21.15.46:0/2617236438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530002cb20 con 0x7f5308142270 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.947+0000 7f530d063640 1 -- 172.21.15.46:0/2617236438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f530002cc80 con 0x7f5308142270 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.947+0000 7f530d063640 1 -- 172.21.15.46:0/2617236438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53000343f0 con 0x7f5308142270 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 -- 172.21.15.46:0/2617236438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5308142270 msgr2=0x7f53081426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 --2- 172.21.15.46:0/2617236438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5308142270 0x7f53081426c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5300009e50 tx=0x7f530002bf80 comp rx=0 tx=0).stop 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 -- 172.21.15.46:0/2617236438 shutdown_connections 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 --2- 172.21.15.46:0/2617236438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5308142270 0x7f53081426c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 -- 172.21.15.46:0/2617236438 >> 172.21.15.46:0/2617236438 conn(0x7f530806a6b0 msgr2=0x7f530806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 -- 172.21.15.46:0/2617236438 shutdown_connections 2023-12-16T21:14:20.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.948+0000 7f53102c9640 1 -- 172.21.15.46:0/2617236438 wait complete. 2023-12-16T21:14:20.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.949+0000 7f53102c9640 1 Processor -- start 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.949+0000 7f53102c9640 1 -- start start 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f53102c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 0x7f53081e7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f53102c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5308142270 0x7f530813dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f53102c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5308160fb0 0x7f530813e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f53102c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530813f670 con 0x7f530806ea50 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f53102c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530813fa70 con 0x7f5308142270 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f53102c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f530813e740 con 0x7f5308160fb0 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 0x7f53081e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 0x7f53081e7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51882/0 (socket says 172.21.15.46:51882) 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 -- 172.21.15.46:0/2725124081 learned_addr learned my addr 172.21.15.46:0/2725124081 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e866640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5308160fb0 0x7f530813e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530d864640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5308142270 0x7f530813dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 -- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5308160fb0 msgr2=0x7f530813e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5308160fb0 0x7f530813e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 -- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5308142270 msgr2=0x7f530813dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5308142270 0x7f530813dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 -- 172.21.15.46:0/2725124081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5300009b30 con 0x7f530806ea50 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.950+0000 7f530e065640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 0x7f53081e7800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f530400c400 tx=0x7f530400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.951+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53040052d0 con 0x7f530806ea50 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.951+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5304005430 con 0x7f530806ea50 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.951+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304003830 con 0x7f530806ea50 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.951+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530813e9f0 con 0x7f530806ea50 2023-12-16T21:14:20.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.951+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530813efb0 con 0x7f530806ea50 2023-12-16T21:14:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.952+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5308069400 con 0x7f530806ea50 2023-12-16T21:14:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.952+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5304003990 con 0x7f530806ea50 2023-12-16T21:14:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.952+0000 7f52feffd640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f52ec06ae70 0x7f52ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.953+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5304088b10 con 0x7f530806ea50 2023-12-16T21:14:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.953+0000 7f530d864640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f52ec06ae70 0x7f52ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:20.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.953+0000 7f530d864640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f52ec06ae70 0x7f52ec06d330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f53080b2870 tx=0x7f52f8000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:20.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:20.957+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5304053bf0 con 0x7f530806ea50 2023-12-16T21:14:21.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.145+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5308070d30 con 0x7f530806ea50 2023-12-16T21:14:21.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.145+0000 7f52feffd640 1 -- 172.21.15.46:0/2725124081 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5304057210 con 0x7f530806ea50 2023-12-16T21:14:21.145 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f52ec06ae70 msgr2=0x7f52ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f52ec06ae70 0x7f52ec06d330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f53080b2870 tx=0x7f52f8000ed0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 msgr2=0x7f53081e7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 0x7f53081e7800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f530400c400 tx=0x7f530400c8c0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 shutdown_connections 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f52ec06ae70 0x7f52ec06d330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5308160fb0 0x7f530813e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5308142270 0x7f530813dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 --2- 172.21.15.46:0/2725124081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f530806ea50 0x7f53081e7800 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 >> 172.21.15.46:0/2725124081 conn(0x7f530806a6b0 msgr2=0x7f530815b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:21.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 shutdown_connections 2023-12-16T21:14:21.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.147+0000 7f53102c9640 1 -- 172.21.15.46:0/2725124081 wait complete. 2023-12-16T21:14:21.158 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:14:21.158 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:21.175 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:21.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.281+0000 7fe04f961640 1 Processor -- start 2023-12-16T21:14:21.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.282+0000 7fe04f961640 1 -- start start 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.282+0000 7fe04f961640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe0480d2910 0x7fe0480d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.282+0000 7fe04f961640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d3730 0x7fe0480b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.282+0000 7fe04f961640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0480b2b10 con 0x7fe04815d330 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.282+0000 7fe04f961640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480d1590 con 0x7fe0480d2910 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.282+0000 7fe04f961640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480d1870 con 0x7fe0480d3730 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04cefc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe0480d2910 0x7fe0480d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04defe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d3730 0x7fe0480b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04d6fd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fe04815d330 0x7fe0480b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:58684/0 (socket says 172.21.15.46:58684) 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04d6fd640 1 -- 172.21.15.46:0/3561507708 learned_addr learned my addr 172.21.15.46:0/3561507708 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04cefc640 1 -- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d3730 msgr2=0x7fe0480b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:21.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04cefc640 1 --2- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d3730 0x7fe0480b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04cefc640 1 -- 172.21.15.46:0/3561507708 >> v1:172.21.15.121:6789/0 conn(0x7fe04815d330 legacy=0x7fe0480b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04cefc640 1 -- 172.21.15.46:0/3561507708 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0480d3b90 con 0x7fe0480d2910 2023-12-16T21:14:21.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe03ffff640 1 -- 172.21.15.46:0/3561507708 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701912186 0 0) 0x7fe0480b2b10 con 0x7fe04815d330 2023-12-16T21:14:21.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.283+0000 7fe04defe640 1 --2- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d3730 0x7fe0480b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04cefc640 1 --2- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0480d2910 0x7fe0480d31f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe0380097a0 tx=0x7fe03802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fed6c6ef91ee9f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe03ffff640 1 -- 172.21.15.46:0/3561507708 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03802e070 con 0x7fe0480d2910 2023-12-16T21:14:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe03ffff640 1 -- 172.21.15.46:0/3561507708 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0380315a0 con 0x7fe0480d2910 2023-12-16T21:14:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe03ffff640 1 -- 172.21.15.46:0/3561507708 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038031930 con 0x7fe0480d2910 2023-12-16T21:14:21.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04f961640 1 -- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0480d2910 msgr2=0x7fe0480d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04f961640 1 --2- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0480d2910 0x7fe0480d31f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe0380097a0 tx=0x7fe03802b840 comp rx=0 tx=0).stop 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04f961640 1 -- 172.21.15.46:0/3561507708 shutdown_connections 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04f961640 1 --2- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d3730 0x7fe0480b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04f961640 1 --2- 172.21.15.46:0/3561507708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0480d2910 0x7fe0480d31f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.284+0000 7fe04f961640 1 -- 172.21.15.46:0/3561507708 >> 172.21.15.46:0/3561507708 conn(0x7fe048072020 msgr2=0x7fe048072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.285+0000 7fe04f961640 1 -- 172.21.15.46:0/3561507708 shutdown_connections 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.285+0000 7fe04f961640 1 -- 172.21.15.46:0/3561507708 wait complete. 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 Processor -- start 2023-12-16T21:14:21.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 -- start start 2023-12-16T21:14:21.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d2910 0x7fe04814e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 0x7fe04814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815d330 0x7fe04815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04815e040 con 0x7fe0480d3730 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480c99e0 con 0x7fe04815d330 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.286+0000 7fe04f961640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04815d710 con 0x7fe0480d2910 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 0x7fe04814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 0x7fe04814f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51894/0 (socket says 172.21.15.46:51894) 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 -- 172.21.15.46:0/3537723971 learned_addr learned my addr 172.21.15.46:0/3537723971 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04d6fd640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d2910 0x7fe04814e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04defe640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815d330 0x7fe04815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 -- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d2910 msgr2=0x7fe04814e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d2910 0x7fe04814e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 -- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815d330 msgr2=0x7fe04815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815d330 0x7fe04815e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 -- 172.21.15.46:0/3537723971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040009a70 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04cefc640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 0x7fe04814f220 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe038034040 tx=0x7fe0380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038032630 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe038032790 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe038009400 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.287+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe038032b20 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.288+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04815dcd0 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.288+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe048069400 con 0x7fe0480d3730 2023-12-16T21:14:21.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.289+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe038032c80 con 0x7fe0480d3730 2023-12-16T21:14:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.289+0000 7fe03dffb640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe02c06ae40 0x7fe02c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.289+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe0380ab580 con 0x7fe0480d3730 2023-12-16T21:14:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.290+0000 7fe04d6fd640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe02c06ae40 0x7fe02c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.290+0000 7fe04d6fd640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe02c06ae40 0x7fe02c06d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe044007fd0 tx=0x7fe044007de0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.293+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe038076490 con 0x7fe0480d3730 2023-12-16T21:14:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.474+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0480b6830 con 0x7fe0480d3730 2023-12-16T21:14:21.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.475+0000 7fe03dffb640 1 -- 172.21.15.46:0/3537723971 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16754 (secure 0 0 0) 0x7fe038079ab0 con 0x7fe0480d3730 2023-12-16T21:14:21.475 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:21.475 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:17.654454+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:21.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe02c06ae40 msgr2=0x7fe02c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe02c06ae40 0x7fe02c06d300 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe044007fd0 tx=0x7fe044007de0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 msgr2=0x7fe04814f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 0x7fe04814f220 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe038034040 tx=0x7fe0380046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 shutdown_connections 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe02c06ae40 0x7fe02c06d300 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0480d3730 0x7fe04814f220 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe04815d330 0x7fe04815e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 --2- 172.21.15.46:0/3537723971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0480d2910 0x7fe04814e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 >> 172.21.15.46:0/3537723971 conn(0x7fe048072020 msgr2=0x7fe048072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 shutdown_connections 2023-12-16T21:14:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.477+0000 7fe04f961640 1 -- 172.21.15.46:0/3537723971 wait complete. 2023-12-16T21:14:21.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:21.488 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.597+0000 7fa8c25b0640 1 Processor -- start 2023-12-16T21:14:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.597+0000 7fa8c25b0640 1 -- start start 2023-12-16T21:14:21.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8c25b0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8bc15b130 0x7fa8bc0d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8c25b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc0cd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8c25b0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc0b3f10 con 0x7fa8bc15f760 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8c25b0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8bc15fd20 con 0x7fa8bc15b130 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8c25b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc0d0730 con 0x7fa8bc0cf320 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8bbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8bc15b130 0x7fa8bc0d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8bbfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa8bc15b130 0x7fa8bc0d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51904/0 (socket says 172.21.15.46:51904) 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.598+0000 7fa8bbfff640 1 -- 172.21.15.46:0/3468035406 learned_addr learned my addr 172.21.15.46:0/3468035406 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.599+0000 7fa8c0b4d640 1 --2- 172.21.15.46:0/3468035406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc0cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.599+0000 7fa8bbfff640 1 -- 172.21.15.46:0/3468035406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 msgr2=0x7fa8bc0cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.599+0000 7fa8bbfff640 1 --2- 172.21.15.46:0/3468035406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc0cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.599+0000 7fa8bbfff640 1 -- 172.21.15.46:0/3468035406 >> v1:172.21.15.121:6789/0 conn(0x7fa8bc15f760 legacy=0x7fa8bc0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.599+0000 7fa8bbfff640 1 -- 172.21.15.46:0/3468035406 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0d0a50 con 0x7fa8bc15b130 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.599+0000 7fa8bbfff640 1 --2- 172.21.15.46:0/3468035406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15b130 0x7fa8bc0d01f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc06ede0 tx=0x7fa8ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6623a4d7c5a88 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8baffd640 1 -- 172.21.15.46:0/3468035406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac02e070 con 0x7fa8bc15b130 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8baffd640 1 -- 172.21.15.46:0/3468035406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ac0315a0 con 0x7fa8bc15b130 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8baffd640 1 -- 172.21.15.46:0/3468035406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac031930 con 0x7fa8bc15b130 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 -- 172.21.15.46:0/3468035406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15b130 msgr2=0x7fa8bc0d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/3468035406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15b130 0x7fa8bc0d01f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc06ede0 tx=0x7fa8ac02b840 comp rx=0 tx=0).stop 2023-12-16T21:14:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 -- 172.21.15.46:0/3468035406 shutdown_connections 2023-12-16T21:14:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/3468035406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc0cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/3468035406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15b130 0x7fa8bc0d01f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 -- 172.21.15.46:0/3468035406 >> 172.21.15.46:0/3468035406 conn(0x7fa8bc0b2c10 msgr2=0x7fa8bc0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 -- 172.21.15.46:0/3468035406 shutdown_connections 2023-12-16T21:14:21.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.600+0000 7fa8c25b0640 1 -- 172.21.15.46:0/3468035406 wait complete. 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.601+0000 7fa8c25b0640 1 Processor -- start 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 -- start start 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc14f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8bc15b130 0x7fa8bc15d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 0x7fa8bc0c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc150950 con 0x7fa8bc15f760 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc15c6e0 con 0x7fa8bc15b130 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c25b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc15c9c0 con 0x7fa8bc0cf320 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c0b4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 0x7fa8bc0c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c0b4d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 0x7fa8bc0c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51910/0 (socket says 172.21.15.46:51910) 2023-12-16T21:14:21.603 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8c0b4d640 1 -- 172.21.15.46:0/2196348190 learned_addr learned my addr 172.21.15.46:0/2196348190 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8bb7fe640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8bc15b130 0x7fa8bc15d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.602+0000 7fa8bbfff640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc14f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c0b4d640 1 -- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 msgr2=0x7fa8bc14f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c0b4d640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc14f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c0b4d640 1 -- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8bc15b130 msgr2=0x7fa8bc15d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c0b4d640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8bc15b130 0x7fa8bc15d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c0b4d640 1 -- 172.21.15.46:0/2196348190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ac0094d0 con 0x7fa8bc15f760 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c0b4d640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 0x7fa8bc0c97e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400cd40 tx=0x7fa8b400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4003b90 con 0x7fa8bc15f760 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4003cf0 con 0x7fa8bc15f760 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8bc15cca0 con 0x7fa8bc15f760 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4004240 con 0x7fa8bc15f760 2023-12-16T21:14:21.604 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.603+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc14fb00 con 0x7fa8bc15f760 2023-12-16T21:14:21.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.604+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8bc067fc0 con 0x7fa8bc15f760 2023-12-16T21:14:21.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.605+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa8b40043a0 con 0x7fa8bc15f760 2023-12-16T21:14:21.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.605+0000 7fa8b8ff9640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa8a806ae40 0x7fa8a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.605+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa8b4012070 con 0x7fa8bc15f760 2023-12-16T21:14:21.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.605+0000 7fa8bbfff640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa8a806ae40 0x7fa8a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.606+0000 7fa8bbfff640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa8a806ae40 0x7fa8a806d300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac036000 tx=0x7fa8ac03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.609+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa8b408e070 con 0x7fa8bc15f760 2023-12-16T21:14:21.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.787+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8bc150a90 con 0x7fa8bc15f760 2023-12-16T21:14:21.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.787+0000 7fa8b8ff9640 1 -- 172.21.15.46:0/2196348190 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8b4054f90 con 0x7fa8bc15f760 2023-12-16T21:14:21.787 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa8a806ae40 msgr2=0x7fa8a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa8a806ae40 0x7fa8a806d300 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac036000 tx=0x7fa8ac03d040 comp rx=0 tx=0).stop 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 msgr2=0x7fa8bc0c97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 0x7fa8bc0c97e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400cd40 tx=0x7fa8b400a660 comp rx=0 tx=0).stop 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 shutdown_connections 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa8a806ae40 0x7fa8a806d300 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa8bc15f760 0x7fa8bc0c97e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8bc15b130 0x7fa8bc15d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 --2- 172.21.15.46:0/2196348190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8bc0cf320 0x7fa8bc14f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 >> 172.21.15.46:0/2196348190 conn(0x7fa8bc0b2c10 msgr2=0x7fa8bc1592a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 shutdown_connections 2023-12-16T21:14:21.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.789+0000 7fa8c25b0640 1 -- 172.21.15.46:0/2196348190 wait complete. 2023-12-16T21:14:21.799 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:21.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.908+0000 7f040cd61640 1 Processor -- start 2023-12-16T21:14:21.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040cd61640 1 -- start start 2023-12-16T21:14:21.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040cd61640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f040815d330 0x7f04080b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040cd61640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04080d2910 0x7f04080d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040cd61640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04080b2b10 con 0x7f04080d3730 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040cd61640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04080d3bc0 con 0x7f040815d330 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040cd61640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04080d1590 con 0x7f04080d2910 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f040815d330 0x7f04080b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f040815d330 0x7f04080b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51912/0 (socket says 172.21.15.46:51912) 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.909+0000 7f040659c640 1 -- 172.21.15.46:0/2255594209 learned_addr learned my addr 172.21.15.46:0/2255594209 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.910+0000 7f03fffff640 1 --2- 172.21.15.46:0/2255594209 >> v2:172.21.15.121:3300/0 conn(0x7f04080d2910 0x7f04080d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.910+0000 7f040659c640 1 -- 172.21.15.46:0/2255594209 >> v2:172.21.15.121:3300/0 conn(0x7f04080d2910 msgr2=0x7f04080d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.910+0000 7f040659c640 1 --2- 172.21.15.46:0/2255594209 >> v2:172.21.15.121:3300/0 conn(0x7f04080d2910 0x7f04080d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.910+0000 7f040659c640 1 -- 172.21.15.46:0/2255594209 >> v1:172.21.15.121:6789/0 conn(0x7f04080d3730 legacy=0x7f04080b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.910+0000 7f040659c640 1 -- 172.21.15.46:0/2255594209 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04080d1870 con 0x7f040815d330 2023-12-16T21:14:21.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040659c640 1 --2- 172.21.15.46:0/2255594209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f040815d330 0x7f04080b8b80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04080d1540 tx=0x7f03f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e1538face53e15 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f0405d9b640 1 -- 172.21.15.46:0/2255594209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8031870 con 0x7f040815d330 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f0405d9b640 1 -- 172.21.15.46:0/2255594209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f80319d0 con 0x7f040815d330 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f0405d9b640 1 -- 172.21.15.46:0/2255594209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8031d60 con 0x7f040815d330 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 -- 172.21.15.46:0/2255594209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f040815d330 msgr2=0x7f04080b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 --2- 172.21.15.46:0/2255594209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f040815d330 0x7f04080b8b80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04080d1540 tx=0x7f03f802b910 comp rx=0 tx=0).stop 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 -- 172.21.15.46:0/2255594209 shutdown_connections 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 --2- 172.21.15.46:0/2255594209 >> v2:172.21.15.121:3300/0 conn(0x7f04080d2910 0x7f04080d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 --2- 172.21.15.46:0/2255594209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f040815d330 0x7f04080b8b80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 -- 172.21.15.46:0/2255594209 >> 172.21.15.46:0/2255594209 conn(0x7f0408072020 msgr2=0x7f0408072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 -- 172.21.15.46:0/2255594209 shutdown_connections 2023-12-16T21:14:21.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.911+0000 7f040cd61640 1 -- 172.21.15.46:0/2255594209 wait complete. 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 Processor -- start 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 -- start start 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04080d2910 0x7f0408156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 0x7f040815fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f040815d330 0x7f04081602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408157d40 con 0x7f04080d3730 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408160810 con 0x7f04080d2910 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f040cd61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04080c8100 con 0x7f040815d330 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.913+0000 7f03fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 0x7f040815fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 0x7f040815fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51926/0 (socket says 172.21.15.46:51926) 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 -- 172.21.15.46:0/3792198746 learned_addr learned my addr 172.21.15.46:0/3792198746 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:21.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f040659c640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04080d2910 0x7f0408156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f0406d9d640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f040815d330 0x7f04081602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 -- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f040815d330 msgr2=0x7f04081602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f040815d330 0x7f04081602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 -- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04080d2910 msgr2=0x7f0408156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04080d2910 0x7f0408156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 -- 172.21.15.46:0/3792198746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80094d0 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03fffff640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 0x7f040815fd90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f03f000d830 tx=0x7f03f000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f000b6c0 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f000b820 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0408157780 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f000bbb0 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.914+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04080c7d90 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.915+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04080c7bf0 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.916+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f03f001e460 con 0x7f04080d3730 2023-12-16T21:14:21.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.916+0000 7f03feffd640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03e806ae40 0x7f03e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:21.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.916+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f03f0089cd0 con 0x7f04080d3730 2023-12-16T21:14:21.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.916+0000 7f040659c640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03e806ae40 0x7f03e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:21.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.917+0000 7f040659c640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03e806ae40 0x7f03e806d300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f03f8036000 tx=0x7f03f803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:21.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:21.920+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f04080c7bf0 con 0x7f04080d3730 2023-12-16T21:14:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.094+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f04080c7bf0 con 0x7f04080d3730 2023-12-16T21:14:22.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.095+0000 7f03feffd640 1 -- 172.21.15.46:0/3792198746 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f04080c7bf0 con 0x7f04080d3730 2023-12-16T21:14:22.095 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03e806ae40 msgr2=0x7f03e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03e806ae40 0x7f03e806d300 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f03f8036000 tx=0x7f03f803c040 comp rx=0 tx=0).stop 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 msgr2=0x7f040815fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 0x7f040815fd90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f03f000d830 tx=0x7f03f000dcf0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 shutdown_connections 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03e806ae40 0x7f03e806d300 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f04080d3730 0x7f040815fd90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f040815d330 0x7f04081602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 --2- 172.21.15.46:0/3792198746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04080d2910 0x7f0408156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 >> 172.21.15.46:0/3792198746 conn(0x7f0408072020 msgr2=0x7f0408072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 shutdown_connections 2023-12-16T21:14:22.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.097+0000 7f040cd61640 1 -- 172.21.15.46:0/3792198746 wait complete. 2023-12-16T21:14:22.108 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:22.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.217+0000 7f4ffed83640 1 Processor -- start 2023-12-16T21:14:22.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.217+0000 7f4ffed83640 1 -- start start 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffed83640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4ff8144a00 0x7f4ff806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffed83640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffed83640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff80b2ef0 con 0x7f4ff81415f0 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffed83640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff80b3b70 con 0x7f4ff8144a00 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffed83640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff8142510 con 0x7f4ff8144440 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffd320640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff7fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4ff8144a00 0x7f4ff806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffcb1f640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4ff81415f0 0x7f4ff8141e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:58718/0 (socket says 172.21.15.46:58718) 2023-12-16T21:14:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffcb1f640 1 -- 172.21.15.46:0/336653391 learned_addr learned my addr 172.21.15.46:0/336653391 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff77fe640 1 -- 172.21.15.46:0/336653391 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409145619 0 0) 0x7f4ff80b2ef0 con 0x7f4ff81415f0 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff77fe640 1 -- 172.21.15.46:0/336653391 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe8003520 con 0x7f4ff81415f0 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff7fff640 1 -- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 msgr2=0x7f4ff806ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff7fff640 1 --2- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff806ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff7fff640 1 -- 172.21.15.46:0/336653391 >> v1:172.21.15.121:6789/0 conn(0x7f4ff81415f0 legacy=0x7f4ff8141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ff7fff640 1 -- 172.21.15.46:0/336653391 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff80b31d0 con 0x7f4ff8144a00 2023-12-16T21:14:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.218+0000 7f4ffd320640 1 --2- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff806ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:22.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.219+0000 7f4ff7fff640 1 --2- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff806b4e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40097d0 tx=0x7f4fe402b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8a0818ba0b3331b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.219+0000 7f4ff77fe640 1 -- 172.21.15.46:0/336653391 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4031800 con 0x7f4ff8144a00 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.219+0000 7f4ff77fe640 1 -- 172.21.15.46:0/336653391 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4031960 con 0x7f4ff8144a00 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.219+0000 7f4ff77fe640 1 -- 172.21.15.46:0/336653391 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4031cd0 con 0x7f4ff8144a00 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 -- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 msgr2=0x7f4ff806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 --2- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff806b4e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40097d0 tx=0x7f4fe402b8a0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 -- 172.21.15.46:0/336653391 shutdown_connections 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 --2- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff806ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 --2- 172.21.15.46:0/336653391 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff806b4e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 -- 172.21.15.46:0/336653391 >> 172.21.15.46:0/336653391 conn(0x7f4ff8070e70 msgr2=0x7f4ff8071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 -- 172.21.15.46:0/336653391 shutdown_connections 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.220+0000 7f4ffed83640 1 -- 172.21.15.46:0/336653391 wait complete. 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.221+0000 7f4ffed83640 1 Processor -- start 2023-12-16T21:14:22.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.221+0000 7f4ffed83640 1 -- start start 2023-12-16T21:14:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffed83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 0x7f4ff806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffed83640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff8140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffed83640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff8156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffed83640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80ad970 con 0x7f4ff81415f0 2023-12-16T21:14:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffed83640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80ad3b0 con 0x7f4ff8144a00 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffed83640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff80ad690 con 0x7f4ff8144440 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffcb1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 0x7f4ff806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffcb1f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 0x7f4ff806d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51938/0 (socket says 172.21.15.46:51938) 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffcb1f640 1 -- 172.21.15.46:0/325851261 learned_addr learned my addr 172.21.15.46:0/325851261 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ffd320640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff8156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.222+0000 7f4ff7fff640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff8140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffcb1f640 1 -- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 msgr2=0x7f4ff8140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffcb1f640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff8140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffcb1f640 1 -- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 msgr2=0x7f4ff8156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffcb1f640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff8156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffcb1f640 1 -- 172.21.15.46:0/325851261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff0009a70 con 0x7f4ff81415f0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffcb1f640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 0x7f4ff806d0b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c430 tx=0x7f4fec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec005ce0 con 0x7f4ff81415f0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec005e40 con 0x7f4ff81415f0 2023-12-16T21:14:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe4009400 con 0x7f4ff81415f0 2023-12-16T21:14:22.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fec0072f0 con 0x7f4ff81415f0 2023-12-16T21:14:22.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.223+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff80ae150 con 0x7f4ff81415f0 2023-12-16T21:14:22.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.224+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff8066ac0 con 0x7f4ff81415f0 2023-12-16T21:14:22.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.224+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4fec007450 con 0x7f4ff81415f0 2023-12-16T21:14:22.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.225+0000 7f4ff57fa640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4fdc06ae40 0x7f4fdc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.225+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4fec088b70 con 0x7f4ff81415f0 2023-12-16T21:14:22.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.225+0000 7f4ff7fff640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4fdc06ae40 0x7f4fdc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.226+0000 7f4ff7fff640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4fdc06ae40 0x7f4fdc06d300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4036000 tx=0x7f4fe403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.229+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4fec053ce0 con 0x7f4ff81415f0 2023-12-16T21:14:22.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.404+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ff8069400 con 0x7f4ff81415f0 2023-12-16T21:14:22.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.404+0000 7f4ff57fa640 1 -- 172.21.15.46:0/325851261 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4fec017020 con 0x7f4ff81415f0 2023-12-16T21:14:22.404 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4fdc06ae40 msgr2=0x7f4fdc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4fdc06ae40 0x7f4fdc06d300 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4036000 tx=0x7f4fe403c040 comp rx=0 tx=0).stop 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 msgr2=0x7f4ff806d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 0x7f4ff806d0b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4fec00c430 tx=0x7f4fec00c830 comp rx=0 tx=0).stop 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 shutdown_connections 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4fdc06ae40 0x7f4fdc06d300 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ff8144a00 0x7f4ff8156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ff8144440 0x7f4ff8140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 --2- 172.21.15.46:0/325851261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ff81415f0 0x7f4ff806d0b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 >> 172.21.15.46:0/325851261 conn(0x7f4ff8070e70 msgr2=0x7f4ff81428c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:22.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 shutdown_connections 2023-12-16T21:14:22.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.406+0000 7f4ffed83640 1 -- 172.21.15.46:0/325851261 wait complete. 2023-12-16T21:14:22.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.525+0000 7f33985d7640 1 Processor -- start 2023-12-16T21:14:22.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.525+0000 7f33985d7640 1 -- start start 2023-12-16T21:14:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f33985d7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3390143490 0x7f33900b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f33985d7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339015e2a0 con 0x7f339015abf0 2023-12-16T21:14:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f33985d7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339015b240 con 0x7f339015e730 2023-12-16T21:14:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f33985d7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f339015b520 con 0x7f3390143490 2023-12-16T21:14:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3396373640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3390143490 0x7f33900b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3396b74640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f339015abf0 0x7f33900b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44974/0 (socket says 172.21.15.46:44974) 2023-12-16T21:14:22.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3396b74640 1 -- 172.21.15.46:0/3877540877 learned_addr learned my addr 172.21.15.46:0/3877540877 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3396373640 1 -- 172.21.15.46:0/3877540877 >> v1:172.21.15.121:6789/0 conn(0x7f339015e730 legacy=0x7f339015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3396373640 1 -- 172.21.15.46:0/3877540877 >> v1:172.21.15.46:6789/0 conn(0x7f339015abf0 legacy=0x7f33900b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3396373640 1 -- 172.21.15.46:0/3877540877 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339015dc50 con 0x7f3390143490 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.526+0000 7f3395371640 1 -- 172.21.15.46:0/3877540877 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173950449 0 0) 0x7f339015e2a0 con 0x7f339015abf0 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.527+0000 7f3396373640 1 --2- 172.21.15.46:0/3877540877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900b0cf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097a0 tx=0x7f338c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f0c76b2da5950d6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.527+0000 7f3395371640 1 -- 172.21.15.46:0/3877540877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c030610 con 0x7f3390143490 2023-12-16T21:14:22.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.527+0000 7f3395371640 1 -- 172.21.15.46:0/3877540877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c030770 con 0x7f3390143490 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.527+0000 7f3395371640 1 -- 172.21.15.46:0/3877540877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c030b00 con 0x7f3390143490 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.527+0000 7f33985d7640 1 -- 172.21.15.46:0/3877540877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 msgr2=0x7f33900b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.527+0000 7f33985d7640 1 --2- 172.21.15.46:0/3877540877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900b0cf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097a0 tx=0x7f338c02b960 comp rx=0 tx=0).stop 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.528+0000 7f33985d7640 1 -- 172.21.15.46:0/3877540877 shutdown_connections 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.528+0000 7f33985d7640 1 --2- 172.21.15.46:0/3877540877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900b0cf0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.528+0000 7f33985d7640 1 -- 172.21.15.46:0/3877540877 >> 172.21.15.46:0/3877540877 conn(0x7f339006a4d0 msgr2=0x7f339006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.528+0000 7f33985d7640 1 -- 172.21.15.46:0/3877540877 shutdown_connections 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.528+0000 7f33985d7640 1 -- 172.21.15.46:0/3877540877 wait complete. 2023-12-16T21:14:22.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.529+0000 7f33985d7640 1 Processor -- start 2023-12-16T21:14:22.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.529+0000 7f33985d7640 1 -- start start 2023-12-16T21:14:22.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f339015abf0 0x7f339015c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f339015e730 0x7f33900cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339015c5b0 con 0x7f3390143490 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339015d3d0 con 0x7f339015abf0 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339015c7c0 con 0x7f339015e730 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900ada10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51952/0 (socket says 172.21.15.46:51952) 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 -- 172.21.15.46:0/3436642893 learned_addr learned my addr 172.21.15.46:0/3436642893 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 -- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f339015e730 msgr2=0x7f33900cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f339015e730 0x7f33900cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3395b72640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f339015abf0 0x7f339015c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 -- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f339015abf0 msgr2=0x7f339015c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f339015abf0 0x7f339015c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 -- 172.21.15.46:0/3436642893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c009400 con 0x7f3390143490 2023-12-16T21:14:22.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3395b72640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f339015abf0 0x7f339015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3396373640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900ada10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f338c0098c0 tx=0x7f338c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c03e040 con 0x7f3390143490 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c039cb0 con 0x7f3390143490 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339015caa0 con 0x7f3390143490 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c044630 con 0x7f3390143490 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.530+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339015d060 con 0x7f3390143490 2023-12-16T21:14:22.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.531+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339015d560 con 0x7f3390143490 2023-12-16T21:14:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.532+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f338c04d3b0 con 0x7f3390143490 2023-12-16T21:14:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.532+0000 7f3386ffd640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f337406ae40 0x7f337406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.532+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f338c0abd40 con 0x7f3390143490 2023-12-16T21:14:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.533+0000 7f3395b72640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f337406ae40 0x7f337406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.533+0000 7f3395b72640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f337406ae40 0x7f337406d300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33900b0360 tx=0x7f3380009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.536+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f338c076cd0 con 0x7f3390143490 2023-12-16T21:14:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.725+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3390069400 con 0x7f3390143490 2023-12-16T21:14:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.726+0000 7f3386ffd640 1 -- 172.21.15.46:0/3436642893 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f338c07a2f0 con 0x7f3390143490 2023-12-16T21:14:22.726 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.727+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f337406ae40 msgr2=0x7f337406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.727+0000 7f33985d7640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f337406ae40 0x7f337406d300 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33900b0360 tx=0x7f3380009040 comp rx=0 tx=0).stop 2023-12-16T21:14:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 msgr2=0x7f33900ada10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900ada10 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f338c0098c0 tx=0x7f338c004730 comp rx=0 tx=0).stop 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 shutdown_connections 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f337406ae40 0x7f337406d300 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f339015e730 0x7f33900cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f339015abf0 0x7f339015c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 --2- 172.21.15.46:0/3436642893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3390143490 0x7f33900ada10 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 >> 172.21.15.46:0/3436642893 conn(0x7f339006a4d0 msgr2=0x7f339006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 shutdown_connections 2023-12-16T21:14:22.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.728+0000 7f33985d7640 1 -- 172.21.15.46:0/3436642893 wait complete. 2023-12-16T21:14:22.738 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2023-12-16T21:14:22.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:22.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-12-16T21:14:22.774 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:22.774 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:14:22.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:22.774 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-16T21:14:22.831 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-16T21:14:22.831 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:22.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.939+0000 7fd7ec16f640 1 Processor -- start 2023-12-16T21:14:22.938 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.939+0000 7fd7ec16f640 1 -- start start 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ec16f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e406ea50 0x7fd7e406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ec16f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7e4160fb0 0x7fd7e4161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ec16f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e4161d20 con 0x7fd7e40b3ff0 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ec16f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e40d1ee0 con 0x7fd7e4160fb0 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ec16f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e406f370 con 0x7fd7e406ea50 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e970a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7e4160fb0 0x7fd7e4161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ea70c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd7e40b3ff0 0x7fd7e40b4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44980/0 (socket says 172.21.15.46:44980) 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7ea70c640 1 -- 172.21.15.46:0/1278670363 learned_addr learned my addr 172.21.15.46:0/1278670363 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e9f0b640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e406ea50 0x7fd7e406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e970a640 1 -- 172.21.15.46:0/1278670363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e406ea50 msgr2=0x7fd7e406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e970a640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e406ea50 0x7fd7e406ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.940 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e970a640 1 -- 172.21.15.46:0/1278670363 >> v1:172.21.15.46:6789/0 conn(0x7fd7e40b3ff0 legacy=0x7fd7e40b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e970a640 1 -- 172.21.15.46:0/1278670363 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e415e760 con 0x7fd7e4160fb0 2023-12-16T21:14:22.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e8f09640 1 -- 172.21.15.46:0/1278670363 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230709123 0 0) 0x7fd7e4161d20 con 0x7fd7e40b3ff0 2023-12-16T21:14:22.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.940+0000 7fd7e9f0b640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e406ea50 0x7fd7e406ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.941+0000 7fd7e970a640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e4160fb0 0x7fd7e4161410 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40097a0 tx=0x7fd7d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af588e7e8086ad1e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.941+0000 7fd7e8f09640 1 -- 172.21.15.46:0/1278670363 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d402e070 con 0x7fd7e4160fb0 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.941+0000 7fd7e8f09640 1 -- 172.21.15.46:0/1278670363 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d40315a0 con 0x7fd7e4160fb0 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.941+0000 7fd7e8f09640 1 -- 172.21.15.46:0/1278670363 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4031930 con 0x7fd7e4160fb0 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 -- 172.21.15.46:0/1278670363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e4160fb0 msgr2=0x7fd7e4161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e4160fb0 0x7fd7e4161410 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40097a0 tx=0x7fd7d402b840 comp rx=0 tx=0).stop 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 -- 172.21.15.46:0/1278670363 shutdown_connections 2023-12-16T21:14:22.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e4160fb0 0x7fd7e4161410 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/1278670363 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e406ea50 0x7fd7e406ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 -- 172.21.15.46:0/1278670363 >> 172.21.15.46:0/1278670363 conn(0x7fd7e406a6b0 msgr2=0x7fd7e406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 -- 172.21.15.46:0/1278670363 shutdown_connections 2023-12-16T21:14:22.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.942+0000 7fd7ec16f640 1 -- 172.21.15.46:0/1278670363 wait complete. 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.943+0000 7fd7ec16f640 1 Processor -- start 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.943+0000 7fd7ec16f640 1 -- start start 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ec16f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 0x7fd7e4156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ec16f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e40b3ff0 0x7fd7e406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ec16f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e4160fb0 0x7fd7e4158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ec16f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e406bad0 con 0x7fd7e406ea50 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ec16f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e406c930 con 0x7fd7e40b3ff0 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ec16f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e406bd20 con 0x7fd7e4160fb0 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 0x7fd7e4156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 0x7fd7e4156830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51960/0 (socket says 172.21.15.46:51960) 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 -- 172.21.15.46:0/2138648876 learned_addr learned my addr 172.21.15.46:0/2138648876 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:22.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7ea70c640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e4160fb0 0x7fd7e4158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e970a640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e40b3ff0 0x7fd7e406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 -- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e4160fb0 msgr2=0x7fd7e4158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e4160fb0 0x7fd7e4158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 -- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e40b3ff0 msgr2=0x7fd7e406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e40b3ff0 0x7fd7e406cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 -- 172.21.15.46:0/2138648876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d4009400 con 0x7fd7e406ea50 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.944+0000 7fd7e9f0b640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 0x7fd7e4156830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000d800 tx=0x7fd7e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.945+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0004030 con 0x7fd7e406ea50 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.945+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e0004190 con 0x7fd7e406ea50 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.945+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e406c000 con 0x7fd7e406ea50 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.945+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e0004520 con 0x7fd7e406ea50 2023-12-16T21:14:22.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.945+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e406c5c0 con 0x7fd7e406ea50 2023-12-16T21:14:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.946+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e415bb30 con 0x7fd7e406ea50 2023-12-16T21:14:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.946+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd7e000b6c0 con 0x7fd7e406ea50 2023-12-16T21:14:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.947+0000 7fd7da7fc640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7c806ae70 0x7fd7c806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.947+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd7e00896b0 con 0x7fd7e406ea50 2023-12-16T21:14:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.947+0000 7fd7e970a640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7c806ae70 0x7fd7c806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:22.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.947+0000 7fd7e970a640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7c806ae70 0x7fd7c806d330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4036000 tx=0x7fd7d403e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:22.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:22.951+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd7e0054790 con 0x7fd7e406ea50 2023-12-16T21:14:23.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.163+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7e4066e90 con 0x7fd7e406ea50 2023-12-16T21:14:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.163+0000 7fd7da7fc640 1 -- 172.21.15.46:0/2138648876 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7fd7e0057db0 con 0x7fd7e406ea50 2023-12-16T21:14:23.163 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:23.163 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:14:23.163 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7c806ae70 msgr2=0x7fd7c806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7c806ae70 0x7fd7c806d330 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4036000 tx=0x7fd7d403e040 comp rx=0 tx=0).stop 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 msgr2=0x7fd7e4156830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 0x7fd7e4156830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd7e000d800 tx=0x7fd7e000dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 shutdown_connections 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7c806ae70 0x7fd7c806d330 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7e4160fb0 0x7fd7e4158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7e40b3ff0 0x7fd7e406cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 --2- 172.21.15.46:0/2138648876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7e406ea50 0x7fd7e4156830 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 >> 172.21.15.46:0/2138648876 conn(0x7fd7e406a6b0 msgr2=0x7fd7e415b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:23.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 shutdown_connections 2023-12-16T21:14:23.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.165+0000 7fd7ec16f640 1 -- 172.21.15.46:0/2138648876 wait complete. 2023-12-16T21:14:23.176 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:23.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.282+0000 7f2053844640 1 Processor -- start 2023-12-16T21:14:23.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.283+0000 7f2053844640 1 -- start start 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.283+0000 7f2053844640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f204c144a00 0x7f204c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.283+0000 7f2053844640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 0x7f204c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2053844640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c0b2eb0 con 0x7f204c1415f0 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2053844640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204c0b3190 con 0x7f204c144a00 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2053844640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0b3470 con 0x7f204c144440 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2051de1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 0x7f204c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f204c144a00 0x7f204c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f204c144a00 0x7f204c06b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:36566/0 (socket says 172.21.15.46:36566) 2023-12-16T21:14:23.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f20515e0640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f204c1415f0 0x7f204c141e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:58728/0 (socket says 172.21.15.46:58728) 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 -- 172.21.15.46:0/835067332 learned_addr learned my addr 172.21.15.46:0/835067332 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2051de1640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 0x7f204c06ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:45982/0 (socket says 172.21.15.46:45982) 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 -- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 msgr2=0x7f204c06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 --2- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 0x7f204c06ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 -- 172.21.15.46:0/835067332 >> v1:172.21.15.121:6789/0 conn(0x7f204c1415f0 legacy=0x7f204c141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2050ddf640 1 -- 172.21.15.46:0/835067332 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c0b3750 con 0x7f204c144a00 2023-12-16T21:14:23.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.284+0000 7f2051de1640 1 --2- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 0x7f204c06ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:23.286 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.285+0000 7f2050ddf640 1 --2- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144a00 0x7f204c06b4e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2040009870 tx=0x7f204002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc61e258ab221937 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.285+0000 7f204bfff640 1 -- 172.21.15.46:0/835067332 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040030840 con 0x7f204c144a00 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.285+0000 7f204bfff640 1 -- 172.21.15.46:0/835067332 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20400309a0 con 0x7f204c144a00 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.285+0000 7f204bfff640 1 -- 172.21.15.46:0/835067332 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2040030d30 con 0x7f204c144a00 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 -- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144a00 msgr2=0x7f204c06b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 --2- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144a00 0x7f204c06b4e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2040009870 tx=0x7f204002b980 comp rx=0 tx=0).stop 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 -- 172.21.15.46:0/835067332 shutdown_connections 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 --2- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144440 0x7f204c06ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 --2- 172.21.15.46:0/835067332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144a00 0x7f204c06b4e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 -- 172.21.15.46:0/835067332 >> 172.21.15.46:0/835067332 conn(0x7f204c070e70 msgr2=0x7f204c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 -- 172.21.15.46:0/835067332 shutdown_connections 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.286+0000 7f2053844640 1 -- 172.21.15.46:0/835067332 wait complete. 2023-12-16T21:14:23.287 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.287+0000 7f2053844640 1 Processor -- start 2023-12-16T21:14:23.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.287+0000 7f2053844640 1 -- start start 2023-12-16T21:14:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2053844640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 0x7f204c154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2053844640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144440 0x7f204c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2053844640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144a00 0x7f204c132fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2053844640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c154920 con 0x7f204c1415f0 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2053844640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c154c00 con 0x7f204c144440 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2053844640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c154ee0 con 0x7f204c144a00 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 0x7f204c154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 0x7f204c154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51972/0 (socket says 172.21.15.46:51972) 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 -- 172.21.15.46:0/1443925428 learned_addr learned my addr 172.21.15.46:0/1443925428 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2051de1640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144a00 0x7f204c132fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f2050ddf640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144440 0x7f204c1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 -- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144a00 msgr2=0x7f204c132fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144a00 0x7f204c132fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.291 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 -- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144440 msgr2=0x7f204c1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144440 0x7f204c1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.288+0000 7f20515e0640 1 -- 172.21.15.46:0/1443925428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044009400 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.289+0000 7f20515e0640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 0x7f204c154290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c780 tx=0x7f203c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.289+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c004d60 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.289+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203c00a640 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.289+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203c00aa70 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.289+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20400094d0 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.289+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c1554a0 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.290+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c069400 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.290+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f203c00abd0 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.291+0000 7f2049ffb640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f203806ae40 0x7f203806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.291+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f203c088860 con 0x7f204c1415f0 2023-12-16T21:14:23.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.291+0000 7f2050ddf640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f203806ae40 0x7f203806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.292+0000 7f2050ddf640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f203806ae40 0x7f203806d300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2040036000 tx=0x7f204003b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.295+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f203c0539d0 con 0x7f204c1415f0 2023-12-16T21:14:23.470 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.470+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f204c15cf50 con 0x7f204c1415f0 2023-12-16T21:14:23.471 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.471+0000 7f2049ffb640 1 -- 172.21.15.46:0/1443925428 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f203c056ff0 con 0x7f204c1415f0 2023-12-16T21:14:23.471 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:23.471 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:17.654454+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f203806ae40 msgr2=0x7f203806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f203806ae40 0x7f203806d300 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2040036000 tx=0x7f204003b040 comp rx=0 tx=0).stop 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 msgr2=0x7f204c154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 0x7f204c154290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f203c00c780 tx=0x7f203c00cb80 comp rx=0 tx=0).stop 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 shutdown_connections 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f203806ae40 0x7f203806d300 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f204c144a00 0x7f204c132fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f204c144440 0x7f204c1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 --2- 172.21.15.46:0/1443925428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f204c1415f0 0x7f204c154290 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 >> 172.21.15.46:0/1443925428 conn(0x7f204c070e70 msgr2=0x7f204c142fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:23.473 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 shutdown_connections 2023-12-16T21:14:23.474 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.473+0000 7f2053844640 1 -- 172.21.15.46:0/1443925428 wait complete. 2023-12-16T21:14:23.484 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:13:34.042927+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:13:34.843424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:23.484 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.595+0000 7f3057957640 1 Processor -- start 2023-12-16T21:14:23.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.595+0000 7f3057957640 1 -- start start 2023-12-16T21:14:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3057957640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f30500d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3057957640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f305015f760 0x7f30500cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.598 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3057957640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500b3f10 con 0x7f30500cf150 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3057957640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305015fd20 con 0x7f305015f760 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3057957640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30500d0730 con 0x7f305015b130 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3055ef4640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f30500cf150 0x7f305006ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44984/0 (socket says 172.21.15.46:44984) 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.596+0000 7f3055ef4640 1 -- 172.21.15.46:0/4177526983 learned_addr learned my addr 172.21.15.46:0/4177526983 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f30556f3640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f30500d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 --2- 172.21.15.46:0/4177526983 >> v2:172.21.15.121:3300/0 conn(0x7f305015f760 0x7f30500cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030165856 0 0) 0x7f30500b3f10 con 0x7f30500cf150 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3038003520 con 0x7f30500cf150 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1653320085 0 0) 0x7f3038003520 con 0x7f30500cf150 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 msgr2=0x7f30500d01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f30500d01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 >> v2:172.21.15.121:3300/0 conn(0x7f305015f760 msgr2=0x7f30500cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 --2- 172.21.15.46:0/4177526983 >> v2:172.21.15.121:3300/0 conn(0x7f305015f760 0x7f30500cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.599 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500d0a50 con 0x7f30500cf150 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f30400019f0 con 0x7f30500cf150 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 >> v1:172.21.15.46:6789/0 conn(0x7f30500cf150 legacy=0x7f305006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3038003970 0x7f3038003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f3038004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f3038005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305015fd20 con 0x7f30380047c0 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500d0730 con 0x7f3038003970 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f303800b980 con 0x7f305015b130 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f3038004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f30556f3640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3038003970 0x7f3038003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3055ef4640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f3038005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 -- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 msgr2=0x7f3038005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f3038005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 -- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3038003970 msgr2=0x7f3038003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.600 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3038003970 0x7f3038003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3054ef2640 1 -- 172.21.15.46:0/4177526983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3038003520 con 0x7f30380047c0 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.597+0000 7f3055ef4640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f305015b130 0x7f3038005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.598+0000 7f3054ef2640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f3038004c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3044009ac0 tx=0x7f30380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da118f3805630ee server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.598+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304402c710 con 0x7f30380047c0 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.598+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304402c870 con 0x7f30380047c0 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.598+0000 7f304ffff640 1 -- 172.21.15.46:0/4177526983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304402cc00 con 0x7f30380047c0 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3057957640 1 -- 172.21.15.46:0/4177526983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 msgr2=0x7f3038004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3057957640 1 --2- 172.21.15.46:0/4177526983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f3038004c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3044009ac0 tx=0x7f30380057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3055ef4640 1 -- 172.21.15.46:0/4177526983 reap_dead start 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3057957640 1 -- 172.21.15.46:0/4177526983 shutdown_connections 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3057957640 1 -- 172.21.15.46:0/4177526983 >> 172.21.15.46:0/4177526983 conn(0x7f30500b2c10 msgr2=0x7f30500b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3057957640 1 -- 172.21.15.46:0/4177526983 shutdown_connections 2023-12-16T21:14:23.601 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.599+0000 7f3057957640 1 -- 172.21.15.46:0/4177526983 wait complete. 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.600+0000 7f3057957640 1 Processor -- start 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.600+0000 7f3057957640 1 -- start start 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3057957640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f305015beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3057957640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30501f4d90 0x7f30500c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3057957640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30501551f0 0x7f3050155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3057957640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500c9970 con 0x7f30380047c0 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3057957640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305015ba90 con 0x7f30501551f0 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3057957640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3050155bb0 con 0x7f30501f4d90 2023-12-16T21:14:23.605 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f305015beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f305015beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51988/0 (socket says 172.21.15.46:51988) 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 -- 172.21.15.46:0/3620247659 learned_addr learned my addr 172.21.15.46:0/3620247659 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3054ef2640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30501f4d90 0x7f30500c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3055ef4640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30501551f0 0x7f3050155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 -- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30501f4d90 msgr2=0x7f30500c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30501f4d90 0x7f30500c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 -- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30501551f0 msgr2=0x7f3050155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30501551f0 0x7f3050155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 -- 172.21.15.46:0/3620247659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500c93b0 con 0x7f30380047c0 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f3054ef2640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30501f4d90 0x7f30500c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.601+0000 7f30556f3640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f305015beb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f30480119a0 tx=0x7f3048011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.602+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304800fa90 con 0x7f30380047c0 2023-12-16T21:14:23.606 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.602+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304800fbf0 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.602+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048014440 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.602+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30500c9690 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.602+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305015c3f0 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.603+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3050067fc0 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.603+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f30480145a0 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.604+0000 7f304dffb640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f303c06ae40 0x7f303c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.604+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3048089da0 con 0x7f30380047c0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.604+0000 7f3054ef2640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f303c06ae40 0x7f303c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.604+0000 7f3054ef2640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f303c06ae40 0x7f303c06d300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3044036000 tx=0x7f3044030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.607 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.608+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3048054e60 con 0x7f30380047c0 2023-12-16T21:14:23.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.784+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f305015b510 con 0x7f30380047c0 2023-12-16T21:14:23.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.784+0000 7f304dffb640 1 -- 172.21.15.46:0/3620247659 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f305015b510 con 0x7f30380047c0 2023-12-16T21:14:23.784 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:23.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f303c06ae40 msgr2=0x7f303c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f303c06ae40 0x7f303c06d300 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3044036000 tx=0x7f3044030000 comp rx=0 tx=0).stop 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 msgr2=0x7f305015beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f305015beb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f30480119a0 tx=0x7f3048011e60 comp rx=0 tx=0).stop 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 shutdown_connections 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f303c06ae40 0x7f303c06d300 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30501551f0 0x7f3050155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30501f4d90 0x7f30500c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 --2- 172.21.15.46:0/3620247659 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30380047c0 0x7f305015beb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 >> 172.21.15.46:0/3620247659 conn(0x7f30500b2c10 msgr2=0x7f305015eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 shutdown_connections 2023-12-16T21:14:23.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.786+0000 7f3057957640 1 -- 172.21.15.46:0/3620247659 wait complete. 2023-12-16T21:14:23.797 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.906+0000 7f68ecce0640 1 Processor -- start 2023-12-16T21:14:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.906+0000 7f68ecce0640 1 -- start start 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.907+0000 7f68ecce0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 0x7f68e815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.907+0000 7f68ecce0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e815e2a0 con 0x7f68e815e730 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.907+0000 7f68ecce0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e815b240 con 0x7f68e8143490 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.907+0000 7f68ecce0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815b520 con 0x7f68e80b1400 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.907+0000 7f68e6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f68e815e730 0x7f68e815aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:44994/0 (socket says 172.21.15.46:44994) 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e6d9d640 1 -- 172.21.15.46:0/1963058290 learned_addr learned my addr 172.21.15.46:0/1963058290 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 0x7f68e815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878110988 0 0) 0x7f68e815e2a0 con 0x7f68e815e730 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68d4003520 con 0x7f68e815e730 2023-12-16T21:14:23.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336945163 0 0) 0x7f68d4003520 con 0x7f68e815e730 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 msgr2=0x7f68e815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 0x7f68e815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 >> v1:172.21.15.121:6789/0 conn(0x7f68e8143490 legacy=0x7f68e80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e815dc50 con 0x7f68e815e730 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f68d80019f0 con 0x7f68e815e730 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 >> v1:172.21.15.46:6789/0 conn(0x7f68e815e730 legacy=0x7f68e815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d40039a0 0x7f68d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 0x7f68d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e815b520 con 0x7f68d40047c0 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d400ba30 con 0x7f68d40039a0 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d400bbf0 con 0x7f68e80b1400 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e6d9d640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 0x7f68d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.908+0000 7f68e659c640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d40039a0 0x7f68d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e5d9b640 1 -- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 msgr2=0x7f68d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80b1400 0x7f68d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e5d9b640 1 -- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d40039a0 msgr2=0x7f68d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68d40039a0 0x7f68d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e5d9b640 1 -- 172.21.15.46:0/1963058290 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4003520 con 0x7f68d40047c0 2023-12-16T21:14:23.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68d4004c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009a90 tx=0x7f68d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255c61ae92b13190 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d002c680 con 0x7f68d40047c0 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.909+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d002c7e0 con 0x7f68d40047c0 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68e559a640 1 -- 172.21.15.46:0/1963058290 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d002cb70 con 0x7f68d40047c0 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68ecce0640 1 -- 172.21.15.46:0/1963058290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 msgr2=0x7f68d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68ecce0640 1 --2- 172.21.15.46:0/1963058290 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68d4004c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009a90 tx=0x7f68d40057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68e6d9d640 1 -- 172.21.15.46:0/1963058290 reap_dead start 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68ecce0640 1 -- 172.21.15.46:0/1963058290 shutdown_connections 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68ecce0640 1 -- 172.21.15.46:0/1963058290 >> 172.21.15.46:0/1963058290 conn(0x7f68e806a4d0 msgr2=0x7f68e806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68ecce0640 1 -- 172.21.15.46:0/1963058290 shutdown_connections 2023-12-16T21:14:23.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.910+0000 7f68ecce0640 1 -- 172.21.15.46:0/1963058290 wait complete. 2023-12-16T21:14:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 Processor -- start 2023-12-16T21:14:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 -- start start 2023-12-16T21:14:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68e80c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e8159af0 0x7f68e80a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80a9350 0x7f68e80a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.916 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8142c10 con 0x7f68d40047c0 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8131200 con 0x7f68e80a9350 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68ecce0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8131aa0 con 0x7f68e8159af0 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68e80c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68e659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68e80c3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52006/0 (socket says 172.21.15.46:52006) 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.912+0000 7f68e659c640 1 -- 172.21.15.46:0/109196063 learned_addr learned my addr 172.21.15.46:0/109196063 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e8159af0 0x7f68e80a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e6d9d640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80a9350 0x7f68e80a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e659c640 1 -- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e8159af0 msgr2=0x7f68e80a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e659c640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e8159af0 0x7f68e80a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e659c640 1 -- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80a9350 msgr2=0x7f68e80a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:23.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e659c640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80a9350 0x7f68e80a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e659c640 1 -- 172.21.15.46:0/109196063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e80abd50 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68e659c640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68e80c3fd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f68e000f850 tx=0x7f68e000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000fe40 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e00052e0 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e8159f70 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e000d970 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.913+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e8131450 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.914+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e8069400 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.915+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f68e000dad0 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.915+0000 7f68deffd640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68c806ae00 0x7f68c806d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.915+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f68e00894a0 con 0x7f68d40047c0 2023-12-16T21:14:23.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.915+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68c806ae00 0x7f68c806d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.916+0000 7f68e5d9b640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68c806ae00 0x7f68c806d2c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68d0036000 tx=0x7f68d0030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:23.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:23.919+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f68e0054580 con 0x7f68d40047c0 2023-12-16T21:14:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.094+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68e80cbe60 con 0x7f68d40047c0 2023-12-16T21:14:24.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.094+0000 7f68deffd640 1 -- 172.21.15.46:0/109196063 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f68e0057ba0 con 0x7f68d40047c0 2023-12-16T21:14:24.094 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:24.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68c806ae00 msgr2=0x7f68c806d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68c806ae00 0x7f68c806d2c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f68d0036000 tx=0x7f68d0030000 comp rx=0 tx=0).stop 2023-12-16T21:14:24.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 msgr2=0x7f68e80c3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68e80c3fd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f68e000f850 tx=0x7f68e000fd10 comp rx=0 tx=0).stop 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 shutdown_connections 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68c806ae00 0x7f68c806d2c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80a9350 0x7f68e80a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e8159af0 0x7f68e80a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 --2- 172.21.15.46:0/109196063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68d40047c0 0x7f68e80c3fd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 >> 172.21.15.46:0/109196063 conn(0x7f68e806a4d0 msgr2=0x7f68e80b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 shutdown_connections 2023-12-16T21:14:24.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.096+0000 7f68ecce0640 1 -- 172.21.15.46:0/109196063 wait complete. 2023-12-16T21:14:24.107 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:24.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.217+0000 7f4850753640 1 Processor -- start 2023-12-16T21:14:24.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.217+0000 7f4850753640 1 -- start start 2023-12-16T21:14:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f4850753640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48481415f0 0x7f4848141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f4850753640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4848144a00 0x7f484806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f4850753640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b2ef0 con 0x7f4848144440 2023-12-16T21:14:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f4850753640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48480b3b70 con 0x7f4848144a00 2023-12-16T21:14:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f4850753640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48480b31d0 con 0x7f48481415f0 2023-12-16T21:14:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4848144a00 0x7f484806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4848144a00 0x7f484806b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52020/0 (socket says 172.21.15.46:52020) 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 -- 172.21.15.46:0/3245403480 learned_addr learned my addr 172.21.15.46:0/3245403480 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484e4ef640 1 --2- 172.21.15.46:0/3245403480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48481415f0 0x7f4848141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 -- 172.21.15.46:0/3245403480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48481415f0 msgr2=0x7f4848141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 --2- 172.21.15.46:0/3245403480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48481415f0 0x7f4848141e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 -- 172.21.15.46:0/3245403480 >> v1:172.21.15.121:6789/0 conn(0x7f4848144440 legacy=0x7f48480b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:24.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.218+0000 7f484dcee640 1 -- 172.21.15.46:0/3245403480 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480b34b0 con 0x7f4848144a00 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.219+0000 7f484dcee640 1 --2- 172.21.15.46:0/3245403480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4848144a00 0x7f484806b4e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4838009870 tx=0x7f483802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a299898391c3003 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.219+0000 7f484d4ed640 1 -- 172.21.15.46:0/3245403480 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838031640 con 0x7f4848144a00 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.219+0000 7f484d4ed640 1 -- 172.21.15.46:0/3245403480 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48380317a0 con 0x7f4848144a00 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.219+0000 7f484d4ed640 1 -- 172.21.15.46:0/3245403480 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4838031b30 con 0x7f4848144a00 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 -- 172.21.15.46:0/3245403480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4848144a00 msgr2=0x7f484806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 --2- 172.21.15.46:0/3245403480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4848144a00 0x7f484806b4e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4838009870 tx=0x7f483802b8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 -- 172.21.15.46:0/3245403480 shutdown_connections 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 --2- 172.21.15.46:0/3245403480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4848144a00 0x7f484806b4e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 --2- 172.21.15.46:0/3245403480 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48481415f0 0x7f4848141e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 -- 172.21.15.46:0/3245403480 >> 172.21.15.46:0/3245403480 conn(0x7f4848070e70 msgr2=0x7f4848071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:24.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 -- 172.21.15.46:0/3245403480 shutdown_connections 2023-12-16T21:14:24.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.220+0000 7f4850753640 1 -- 172.21.15.46:0/3245403480 wait complete. 2023-12-16T21:14:24.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.221+0000 7f4850753640 1 Processor -- start 2023-12-16T21:14:24.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.221+0000 7f4850753640 1 -- start start 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f4850753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 0x7f484815a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f4850753640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4848144440 0x7f4848138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f4850753640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4848144a00 0x7f4848156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f4850753640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484815a7d0 con 0x7f48481415f0 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f4850753640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4848156500 con 0x7f4848144440 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f4850753640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4848138650 con 0x7f4848144a00 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 0x7f484815a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 0x7f484815a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52024/0 (socket says 172.21.15.46:52024) 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 -- 172.21.15.46:0/1818110200 learned_addr learned my addr 172.21.15.46:0/1818110200 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484dcee640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4848144440 0x7f4848138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484ecf0640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4848144a00 0x7f4848156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 -- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4848144a00 msgr2=0x7f4848156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4848144a00 0x7f4848156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 -- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4848144440 msgr2=0x7f4848138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4848144440 0x7f4848138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 -- 172.21.15.46:0/1818110200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48380094d0 con 0x7f48481415f0 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484dcee640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4848144440 0x7f4848138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484ecf0640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4848144a00 0x7f4848156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.222+0000 7f484e4ef640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 0x7f484815a290 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f484000d800 tx=0x7f484000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.223+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000b6c0 con 0x7f48481415f0 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.223+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484000b820 con 0x7f48481415f0 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.223+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48481558f0 con 0x7f48481415f0 2023-12-16T21:14:24.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.223+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484000bbb0 con 0x7f48481415f0 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.223+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4848155eb0 con 0x7f48481415f0 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.224+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4848069400 con 0x7f48481415f0 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.224+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f484000bd10 con 0x7f48481415f0 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.225+0000 7f483effd640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f482c06ae70 0x7f482c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.225+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4840089b50 con 0x7f48481415f0 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.225+0000 7f484dcee640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f482c06ae70 0x7f482c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.225+0000 7f484dcee640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f482c06ae70 0x7f482c06d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4838036000 tx=0x7f483803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.229+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4840054c10 con 0x7f48481415f0 2023-12-16T21:14:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.404+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48480cea80 con 0x7f48481415f0 2023-12-16T21:14:24.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.404+0000 7f483effd640 1 -- 172.21.15.46:0/1818110200 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4840058230 con 0x7f48481415f0 2023-12-16T21:14:24.404 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f482c06ae70 msgr2=0x7f482c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f482c06ae70 0x7f482c06d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4838036000 tx=0x7f483803c040 comp rx=0 tx=0).stop 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 msgr2=0x7f484815a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 0x7f484815a290 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f484000d800 tx=0x7f484000dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 shutdown_connections 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f482c06ae70 0x7f482c06d330 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4848144a00 0x7f4848156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4848144440 0x7f4848138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 --2- 172.21.15.46:0/1818110200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48481415f0 0x7f484815a290 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 >> 172.21.15.46:0/1818110200 conn(0x7f4848070e70 msgr2=0x7f484815f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:24.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 shutdown_connections 2023-12-16T21:14:24.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.406+0000 7f4850753640 1 -- 172.21.15.46:0/1818110200 wait complete. 2023-12-16T21:14:24.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:24.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.525+0000 7f66ad33a640 1 Processor -- start 2023-12-16T21:14:24.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.525+0000 7f66ad33a640 1 -- start start 2023-12-16T21:14:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66ad33a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66a8160fb0 0x7f66a8161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66ad33a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a8161d20 con 0x7f66a806ea50 2023-12-16T21:14:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66ad33a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a815e760 con 0x7f66a80d0ed0 2023-12-16T21:14:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66ad33a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a815d9a0 con 0x7f66a8160fb0 2023-12-16T21:14:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f66a806ea50 0x7f66a806ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45004/0 (socket says 172.21.15.46:45004) 2023-12-16T21:14:24.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a6ffd640 1 -- 172.21.15.46:0/211905948 learned_addr learned my addr 172.21.15.46:0/211905948 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f669e7fc640 1 --2- 172.21.15.46:0/211905948 >> v2:172.21.15.121:3300/0 conn(0x7f66a8160fb0 0x7f66a8161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486649485 0 0) 0x7f66a8161d20 con 0x7f66a806ea50 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694003520 con 0x7f66a806ea50 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266910713 0 0) 0x7f6694003520 con 0x7f66a806ea50 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 >> v2:172.21.15.121:3300/0 conn(0x7f66a8160fb0 msgr2=0x7f66a8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 --2- 172.21.15.46:0/211905948 >> v2:172.21.15.121:3300/0 conn(0x7f66a8160fb0 0x7f66a8161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 >> v1:172.21.15.121:6789/0 conn(0x7f66a80d0ed0 legacy=0x7f66a80d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a815dc80 con 0x7f66a806ea50 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.526+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f6698001670 con 0x7f66a806ea50 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 >> v1:172.21.15.46:6789/0 conn(0x7f66a806ea50 legacy=0x7f66a806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6694005180 0x7f669400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a815d9a0 con 0x7f66940047c0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669400c120 con 0x7f6694005180 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669400c2e0 con 0x7f6694003970 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f669e7fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a6ffd640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a77fe640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6694005180 0x7f669400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f669e7fc640 1 -- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6694003970 msgr2=0x7f6694003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f669e7fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f669e7fc640 1 -- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6694005180 msgr2=0x7f669400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f669e7fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6694005180 0x7f669400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f669e7fc640 1 -- 172.21.15.46:0/211905948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694003520 con 0x7f66940047c0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.527+0000 7f66a77fe640 1 -- 172.21.15.46:0/211905948 reap_dead start 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f669e7fc640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66940047c0 0x7f6694004c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6690009ac0 tx=0x7f66940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8d042207e3ab26 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669002c6b0 con 0x7f66940047c0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669002c810 con 0x7f66940047c0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66a67fc640 1 -- 172.21.15.46:0/211905948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669002cba0 con 0x7f66940047c0 2023-12-16T21:14:24.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66ad33a640 1 -- 172.21.15.46:0/211905948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66940047c0 msgr2=0x7f6694004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66ad33a640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66940047c0 0x7f6694004c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6690009ac0 tx=0x7f66940057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66ad33a640 1 -- 172.21.15.46:0/211905948 shutdown_connections 2023-12-16T21:14:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66ad33a640 1 --2- 172.21.15.46:0/211905948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.528+0000 7f66ad33a640 1 -- 172.21.15.46:0/211905948 >> 172.21.15.46:0/211905948 conn(0x7f66a806a6b0 msgr2=0x7f66a806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.529+0000 7f66ad33a640 1 -- 172.21.15.46:0/211905948 shutdown_connections 2023-12-16T21:14:24.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.529+0000 7f66ad33a640 1 -- 172.21.15.46:0/211905948 wait complete. 2023-12-16T21:14:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.530+0000 7f66ad33a640 1 Processor -- start 2023-12-16T21:14:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.530+0000 7f66ad33a640 1 -- start start 2023-12-16T21:14:24.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66940047c0 0x7f66a81583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 0x7f66a80b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a813be50 0x7f66a80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a81f6ab0 con 0x7f66a813cd10 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a8158d50 con 0x7f66940047c0 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a81f6460 con 0x7f66a813be50 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 0x7f66a80b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 0x7f66a80b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52044/0 (socket says 172.21.15.46:52044) 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 -- 172.21.15.46:0/3441070488 learned_addr learned my addr 172.21.15.46:0/3441070488 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66a6ffd640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66940047c0 0x7f66a81583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66a77fe640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a813be50 0x7f66a80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 -- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a813be50 msgr2=0x7f66a80b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a813be50 0x7f66a80b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 -- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66940047c0 msgr2=0x7f66a81583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66940047c0 0x7f66a81583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 -- 172.21.15.46:0/3441070488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a81f6740 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669e7fc640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 0x7f66a80b1be0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6690009ac0 tx=0x7f6690004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690003810 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6690003970 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a8159030 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690003d00 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.531+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a8158b00 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.532+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a81f68d0 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.533+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6690040450 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.533+0000 7f669ffff640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f668806ae40 0x7f668806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.533+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f66900ab480 con 0x7f66a813cd10 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.534+0000 7f66a6ffd640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f668806ae40 0x7f668806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.534+0000 7f66a6ffd640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f668806ae40 0x7f668806d300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6698007d20 tx=0x7f6698000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.538+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f66900765f0 con 0x7f66a813cd10 2023-12-16T21:14:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.750+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66a8069400 con 0x7f66a813cd10 2023-12-16T21:14:24.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.751+0000 7f669ffff640 1 -- 172.21.15.46:0/3441070488 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7f66900044e0 con 0x7f66a813cd10 2023-12-16T21:14:24.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:24.751 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:14:24.751 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f668806ae40 msgr2=0x7f668806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f668806ae40 0x7f668806d300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6698007d20 tx=0x7f6698000ed0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 msgr2=0x7f66a80b1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 0x7f66a80b1be0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6690009ac0 tx=0x7f6690004730 comp rx=0 tx=0).stop 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 shutdown_connections 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f668806ae40 0x7f668806d300 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f66a813cd10 0x7f66a80b1be0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a813be50 0x7f66a80b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 --2- 172.21.15.46:0/3441070488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66940047c0 0x7f66a81583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 >> 172.21.15.46:0/3441070488 conn(0x7f66a806a6b0 msgr2=0x7f66a80b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 shutdown_connections 2023-12-16T21:14:24.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.753+0000 7f66ad33a640 1 -- 172.21.15.46:0/3441070488 wait complete. 2023-12-16T21:14:24.764 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.871+0000 7f1a15b61640 1 Processor -- start 2023-12-16T21:14:24.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.871+0000 7f1a15b61640 1 -- start start 2023-12-16T21:14:24.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a15b61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a10160be0 0x7f1a100cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a15b61640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a100d0080 0x7f1a100ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a15b61640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100d2230 con 0x7f1a100b1f00 2023-12-16T21:14:24.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a15b61640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a100d0510 con 0x7f1a100d0080 2023-12-16T21:14:24.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a15b61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10160fc0 con 0x7f1a10160be0 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a0ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a100d0080 0x7f1a100ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a0ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a100d0080 0x7f1a100ced90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52054/0 (socket says 172.21.15.46:52054) 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a0ffff640 1 -- 172.21.15.46:0/375887812 learned_addr learned my addr 172.21.15.46:0/375887812 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.872+0000 7f1a0effd640 1 --1- 172.21.15.46:0/375887812 >> v1:172.21.15.46:6789/0 conn(0x7f1a100b1f00 0x7f1a100b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45010/0 (socket says 172.21.15.46:45010) 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.873+0000 7f1a0f7fe640 1 --2- 172.21.15.46:0/375887812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a10160be0 0x7f1a100cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.873+0000 7f1a0ffff640 1 -- 172.21.15.46:0/375887812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a10160be0 msgr2=0x7f1a100cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.873+0000 7f1a0ffff640 1 --2- 172.21.15.46:0/375887812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a10160be0 0x7f1a100cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.873+0000 7f1a0ffff640 1 -- 172.21.15.46:0/375887812 >> v1:172.21.15.46:6789/0 conn(0x7f1a100b1f00 legacy=0x7f1a100b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.873+0000 7f1a0e7fc640 1 -- 172.21.15.46:0/375887812 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872240264 0 0) 0x7f1a100d2230 con 0x7f1a100b1f00 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.873+0000 7f1a0ffff640 1 -- 172.21.15.46:0/375887812 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100d1660 con 0x7f1a100d0080 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a0ffff640 1 --2- 172.21.15.46:0/375887812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100ced90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009e50 tx=0x7f1a0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2fd6a7ccaed0bc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a0e7fc640 1 -- 172.21.15.46:0/375887812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0802cb00 con 0x7f1a100d0080 2023-12-16T21:14:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a0e7fc640 1 -- 172.21.15.46:0/375887812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0802cc60 con 0x7f1a100d0080 2023-12-16T21:14:24.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a0e7fc640 1 -- 172.21.15.46:0/375887812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a080343f0 con 0x7f1a100d0080 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 -- 172.21.15.46:0/375887812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 msgr2=0x7f1a100ced90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 --2- 172.21.15.46:0/375887812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100ced90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009e50 tx=0x7f1a0802bf60 comp rx=0 tx=0).stop 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 -- 172.21.15.46:0/375887812 shutdown_connections 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 --2- 172.21.15.46:0/375887812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100ced90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 --2- 172.21.15.46:0/375887812 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a10160be0 0x7f1a100cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 -- 172.21.15.46:0/375887812 >> 172.21.15.46:0/375887812 conn(0x7f1a1009c480 msgr2=0x7f1a100d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:24.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 -- 172.21.15.46:0/375887812 shutdown_connections 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.874+0000 7f1a15b61640 1 -- 172.21.15.46:0/375887812 wait complete. 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.875+0000 7f1a15b61640 1 Processor -- start 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 -- start start 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a100b1f00 0x7f1a10153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100b7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a10160be0 0x7f1a100ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a101f3e50 con 0x7f1a100d0080 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a101f31d0 con 0x7f1a10160be0 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a15b61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a101f34b0 con 0x7f1a100b1f00 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.876+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100b7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52062/0 (socket says 172.21.15.46:52062) 2023-12-16T21:14:24.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 -- 172.21.15.46:0/3210484148 learned_addr learned my addr 172.21.15.46:0/3210484148 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0f7fe640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a100b1f00 0x7f1a10153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0ffff640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a10160be0 0x7f1a100ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 -- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a100b1f00 msgr2=0x7f1a10153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a100b1f00 0x7f1a10153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 -- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a10160be0 msgr2=0x7f1a100ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a10160be0 0x7f1a100ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 -- 172.21.15.46:0/3210484148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a08009b30 con 0x7f1a100d0080 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a0effd640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100b7560 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c4d0 tx=0x7f1a0400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040074e0 con 0x7f1a100d0080 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a04007660 con 0x7f1a100d0080 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a101f3790 con 0x7f1a100d0080 2023-12-16T21:14:24.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04004a20 con 0x7f1a100d0080 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.877+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a10153a40 con 0x7f1a100d0080 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.878+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a101f3f90 con 0x7f1a100d0080 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.879+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1a04004b80 con 0x7f1a100d0080 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.879+0000 7f19f7fff640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19fc06ae40 0x7f19fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.879+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1a04088b20 con 0x7f1a100d0080 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.879+0000 7f1a0f7fe640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19fc06ae40 0x7f19fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:24.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.880+0000 7f1a0f7fe640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19fc06ae40 0x7f19fc06d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009aa0 tx=0x7f1a00009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:24.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:24.883+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1a101f3f90 con 0x7f1a100d0080 2023-12-16T21:14:25.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.062+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a101f3f90 con 0x7f1a100d0080 2023-12-16T21:14:25.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.063+0000 7f19f7fff640 1 -- 172.21.15.46:0/3210484148 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f1a101f3f90 con 0x7f1a100d0080 2023-12-16T21:14:25.063 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:25.063 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:17.654454+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19fc06ae40 msgr2=0x7f19fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19fc06ae40 0x7f19fc06d300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009aa0 tx=0x7f1a00009190 comp rx=0 tx=0).stop 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 msgr2=0x7f1a100b7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100b7560 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c4d0 tx=0x7f1a0400c990 comp rx=0 tx=0).stop 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 shutdown_connections 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19fc06ae40 0x7f19fc06d300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a100d0080 0x7f1a100b7560 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a10160be0 0x7f1a100ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 --2- 172.21.15.46:0/3210484148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a100b1f00 0x7f1a10153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 >> 172.21.15.46:0/3210484148 conn(0x7f1a1009c480 msgr2=0x7f1a1006b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 shutdown_connections 2023-12-16T21:14:25.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.065+0000 7f1a15b61640 1 -- 172.21.15.46:0/3210484148 wait complete. 2023-12-16T21:14:25.075 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-16T21:14:25.076 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.182+0000 7fdd06a36640 1 Processor -- start 2023-12-16T21:14:25.182 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.182+0000 7fdd06a36640 1 -- start start 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdd06a36640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd00160fb0 0x7fdd00161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdd06a36640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd000d0ed0 0x7fdd000d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdd06a36640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd00161d20 con 0x7fdd0006ea50 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdd06a36640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0015e760 con 0x7fdd00160fb0 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdd06a36640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd0015d9a0 con 0x7fdd000d0ed0 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdcff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd00160fb0 0x7fdd00161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdcff7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd00160fb0 0x7fdd00161440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52078/0 (socket says 172.21.15.46:52078) 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdcff7fe640 1 -- 172.21.15.46:0/2118143757 learned_addr learned my addr 172.21.15.46:0/2118143757 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.183+0000 7fdd04fd3640 1 --2- 172.21.15.46:0/2118143757 >> v2:172.21.15.121:3300/0 conn(0x7fdd000d0ed0 0x7fdd000d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcff7fe640 1 -- 172.21.15.46:0/2118143757 >> v2:172.21.15.121:3300/0 conn(0x7fdd000d0ed0 msgr2=0x7fdd000d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcff7fe640 1 --2- 172.21.15.46:0/2118143757 >> v2:172.21.15.121:3300/0 conn(0x7fdd000d0ed0 0x7fdd000d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcff7fe640 1 -- 172.21.15.46:0/2118143757 >> v1:172.21.15.121:6789/0 conn(0x7fdd0006ea50 legacy=0x7fdd0006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcff7fe640 1 -- 172.21.15.46:0/2118143757 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0015dc80 con 0x7fdd00160fb0 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcff7fe640 1 --2- 172.21.15.46:0/2118143757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00160fb0 0x7fdd00161440 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdd000b55a0 tx=0x7fdcf002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa745b17f0436ffa server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcfeffd640 1 -- 172.21.15.46:0/2118143757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0031870 con 0x7fdd00160fb0 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcfeffd640 1 -- 172.21.15.46:0/2118143757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf00319d0 con 0x7fdd00160fb0 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.184+0000 7fdcfeffd640 1 -- 172.21.15.46:0/2118143757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0031d60 con 0x7fdd00160fb0 2023-12-16T21:14:25.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 -- 172.21.15.46:0/2118143757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00160fb0 msgr2=0x7fdd00161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 --2- 172.21.15.46:0/2118143757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00160fb0 0x7fdd00161440 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdd000b55a0 tx=0x7fdcf002b910 comp rx=0 tx=0).stop 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 -- 172.21.15.46:0/2118143757 shutdown_connections 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 --2- 172.21.15.46:0/2118143757 >> v2:172.21.15.121:3300/0 conn(0x7fdd000d0ed0 0x7fdd000d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 --2- 172.21.15.46:0/2118143757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd00160fb0 0x7fdd00161440 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 -- 172.21.15.46:0/2118143757 >> 172.21.15.46:0/2118143757 conn(0x7fdd0006a6b0 msgr2=0x7fdd0006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 -- 172.21.15.46:0/2118143757 shutdown_connections 2023-12-16T21:14:25.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.185+0000 7fdd06a36640 1 -- 172.21.15.46:0/2118143757 wait complete. 2023-12-16T21:14:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.186+0000 7fdd06a36640 1 Processor -- start 2023-12-16T21:14:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.186+0000 7fdd06a36640 1 -- start start 2023-12-16T21:14:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd06a36640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd0006ea50 0x7fdd0015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd06a36640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 0x7fdd0013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd06a36640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd00160fb0 0x7fdd001f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd06a36640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000b4540 con 0x7fdd000d0ed0 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd06a36640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0015aa90 con 0x7fdd00160fb0 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd06a36640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd000b3e80 con 0x7fdd0006ea50 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdcff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 0x7fdd0013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdcff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 0x7fdd0013ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52080/0 (socket says 172.21.15.46:52080) 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdcff7fe640 1 -- 172.21.15.46:0/3498902279 learned_addr learned my addr 172.21.15.46:0/3498902279 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.187+0000 7fdd04fd3640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd00160fb0 0x7fdd001f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcfffff640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd0006ea50 0x7fdd0015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcff7fe640 1 -- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd0006ea50 msgr2=0x7fdd0015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcff7fe640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd0006ea50 0x7fdd0015a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcff7fe640 1 -- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd00160fb0 msgr2=0x7fdd001f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcff7fe640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd00160fb0 0x7fdd001f6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcff7fe640 1 -- 172.21.15.46:0/3498902279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf00094d0 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdd04fd3640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd00160fb0 0x7fdd001f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcfffff640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd0006ea50 0x7fdd0015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcff7fe640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 0x7fdd0013ba30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdcf002b950 tx=0x7fdcf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf003bcc0 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcf003be40 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd000b4160 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0045630 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.188+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0013bf70 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.189+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd0015b290 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.189+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdcf004e450 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.190+0000 7fdcfcff9640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdce006ae40 0x7fdce006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.190+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdcf002d040 con 0x7fdd000d0ed0 2023-12-16T21:14:25.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.190+0000 7fdcfffff640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdce006ae40 0x7fdce006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.191+0000 7fdcfffff640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdce006ae40 0x7fdce006d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdce8000f60 tx=0x7fdce8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.194+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdcf0076d30 con 0x7fdd000d0ed0 2023-12-16T21:14:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.404+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd00069400 con 0x7fdd000d0ed0 2023-12-16T21:14:25.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.404+0000 7fdcfcff9640 1 -- 172.21.15.46:0/3498902279 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7fdcf0035020 con 0x7fdd000d0ed0 2023-12-16T21:14:25.404 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:25.404 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:14:25.405 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdce006ae40 msgr2=0x7fdce006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdce006ae40 0x7fdce006d300 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fdce8000f60 tx=0x7fdce8009040 comp rx=0 tx=0).stop 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 msgr2=0x7fdd0013ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 0x7fdd0013ba30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdcf002b950 tx=0x7fdcf00046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 shutdown_connections 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdce006ae40 0x7fdce006d300 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd000d0ed0 0x7fdd0013ba30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd00160fb0 0x7fdd001f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 --2- 172.21.15.46:0/3498902279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd0006ea50 0x7fdd0015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 >> 172.21.15.46:0/3498902279 conn(0x7fdd0006a6b0 msgr2=0x7fdd000b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:25.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 shutdown_connections 2023-12-16T21:14:25.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.406+0000 7fdd06a36640 1 -- 172.21.15.46:0/3498902279 wait complete. 2023-12-16T21:14:25.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:25.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.524+0000 7efed059e640 1 Processor -- start 2023-12-16T21:14:25.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.524+0000 7efed059e640 1 -- start start 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efed059e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efec8150940 0x7efec815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efed059e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efec815dec0 0x7efec815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efed059e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec80b49f0 con 0x7efec8150560 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efed059e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec80b3560 con 0x7efec815dec0 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efed059e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efec80b3de0 con 0x7efec8150940 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efeceb3b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efec815dec0 0x7efec815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efece33a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efec8150560 0x7efec80b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45020/0 (socket says 172.21.15.46:45020) 2023-12-16T21:14:25.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efece33a640 1 -- 172.21.15.46:0/1319446 learned_addr learned my addr 172.21.15.46:0/1319446 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:25.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efecdb39640 1 --2- 172.21.15.46:0/1319446 >> v2:172.21.15.121:3300/0 conn(0x7efec8150940 0x7efec815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efeceb3b640 1 -- 172.21.15.46:0/1319446 >> v2:172.21.15.121:3300/0 conn(0x7efec8150940 msgr2=0x7efec815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:25.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efeceb3b640 1 --2- 172.21.15.46:0/1319446 >> v2:172.21.15.121:3300/0 conn(0x7efec8150940 0x7efec815d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efeceb3b640 1 -- 172.21.15.46:0/1319446 >> v1:172.21.15.46:6789/0 conn(0x7efec8150560 legacy=0x7efec80b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efecd338640 1 -- 172.21.15.46:0/1319446 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467305153 0 0) 0x7efec80b49f0 con 0x7efec8150560 2023-12-16T21:14:25.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.525+0000 7efeceb3b640 1 -- 172.21.15.46:0/1319446 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec80b40c0 con 0x7efec815dec0 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.526+0000 7efeceb3b640 1 --2- 172.21.15.46:0/1319446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec815dec0 0x7efec815d740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efec8072810 tx=0x7efec002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f18a807db58698 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.526+0000 7efecd338640 1 -- 172.21.15.46:0/1319446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec002cc80 con 0x7efec815dec0 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.526+0000 7efecd338640 1 -- 172.21.15.46:0/1319446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec002cde0 con 0x7efec815dec0 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.526+0000 7efecd338640 1 -- 172.21.15.46:0/1319446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0034660 con 0x7efec815dec0 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 -- 172.21.15.46:0/1319446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec815dec0 msgr2=0x7efec815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 --2- 172.21.15.46:0/1319446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec815dec0 0x7efec815d740 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efec8072810 tx=0x7efec002bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 -- 172.21.15.46:0/1319446 shutdown_connections 2023-12-16T21:14:25.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 --2- 172.21.15.46:0/1319446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec815dec0 0x7efec815d740 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 --2- 172.21.15.46:0/1319446 >> v2:172.21.15.121:3300/0 conn(0x7efec8150940 0x7efec815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 -- 172.21.15.46:0/1319446 >> 172.21.15.46:0/1319446 conn(0x7efec806a4d0 msgr2=0x7efec806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:25.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 -- 172.21.15.46:0/1319446 shutdown_connections 2023-12-16T21:14:25.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.527+0000 7efed059e640 1 -- 172.21.15.46:0/1319446 wait complete. 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.528+0000 7efed059e640 1 Processor -- start 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.528+0000 7efed059e640 1 -- start start 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efed059e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efec8150560 0x7efec814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efed059e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efec815dec0 0x7efec8159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efed059e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 0x7efec8134470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efed059e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec814ea90 con 0x7efec81475e0 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efed059e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec8157cf0 con 0x7efec815dec0 2023-12-16T21:14:25.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efed059e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec8157fd0 con 0x7efec8150560 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 0x7efec8134470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 0x7efec8134470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52088/0 (socket says 172.21.15.46:52088) 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 -- 172.21.15.46:0/2292583321 learned_addr learned my addr 172.21.15.46:0/2292583321 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efecdb39640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efec815dec0 0x7efec8159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efece33a640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efec8150560 0x7efec814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 -- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efec8150560 msgr2=0x7efec814e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efec8150560 0x7efec814e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 -- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efec815dec0 msgr2=0x7efec8159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efec815dec0 0x7efec8159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.529+0000 7efeceb3b640 1 -- 172.21.15.46:0/2292583321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0009a70 con 0x7efec81475e0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.530+0000 7efeceb3b640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 0x7efec8134470 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efec0009e80 tx=0x7efec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.530+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0040650 con 0x7efec81475e0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.530+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efec00407b0 con 0x7efec81475e0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.530+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec81582b0 con 0x7efec81475e0 2023-12-16T21:14:25.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.530+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0040b60 con 0x7efec81475e0 2023-12-16T21:14:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.530+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec8158870 con 0x7efec81475e0 2023-12-16T21:14:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.531+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec8069400 con 0x7efec81475e0 2023-12-16T21:14:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.531+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7efec0040cc0 con 0x7efec81475e0 2023-12-16T21:14:25.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.532+0000 7efebeffd640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efeac06ae40 0x7efeac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.532+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efec00ac360 con 0x7efec81475e0 2023-12-16T21:14:25.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.532+0000 7efece33a640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efeac06ae40 0x7efeac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.533+0000 7efece33a640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efeac06ae40 0x7efeac06d300 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efec4007fd0 tx=0x7efec4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.536+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7efec0077390 con 0x7efec81475e0 2023-12-16T21:14:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.748+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efec8067420 con 0x7efec81475e0 2023-12-16T21:14:25.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.749+0000 7efebeffd640 1 -- 172.21.15.46:0/2292583321 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4938 (secure 0 0 0) 0x7efec0009450 con 0x7efec81475e0 2023-12-16T21:14:25.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:25.749 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1105704963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1105704963},{"type":"v1","addr":"172.21.15.121:6836","nonce":1105704963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1549588768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1549588768},{"type":"v1","addr":"172.21.15.46:6834","nonce":1549588768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4278,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2488346708","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":2488346708},{"type":"v1","addr":"172.21.15.46:6835","nonce":2488346708}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:13:35.861120+0000","modified":"2023-12-16T21:13:37.862845+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1967157376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1967157376},{"type":"v1","addr":"172.21.15.121:6837","nonce":1967157376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-16T21:14:25.749 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efeac06ae40 msgr2=0x7efeac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efeac06ae40 0x7efeac06d300 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efec4007fd0 tx=0x7efec4009040 comp rx=0 tx=0).stop 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 msgr2=0x7efec8134470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 0x7efec8134470 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efec0009e80 tx=0x7efec0004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 shutdown_connections 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efeac06ae40 0x7efeac06d300 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec81475e0 0x7efec8134470 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efec815dec0 0x7efec8159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 --2- 172.21.15.46:0/2292583321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efec8150560 0x7efec814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 >> 172.21.15.46:0/2292583321 conn(0x7efec806a4d0 msgr2=0x7efec8070400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:25.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 shutdown_connections 2023-12-16T21:14:25.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.751+0000 7efed059e640 1 -- 172.21.15.46:0/2292583321 wait complete. 2023-12-16T21:14:25.761 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.868+0000 7f9a6eada640 1 Processor -- start 2023-12-16T21:14:25.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.868+0000 7f9a6eada640 1 -- start start 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6eada640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a68142270 0x7f9a681426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6eada640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a68161d20 con 0x7f9a6806ea50 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6eada640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6815e760 con 0x7f9a68160fb0 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6eada640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a680d1ee0 con 0x7f9a68142270 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6d077640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f9a68142270 0x7f9a681426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6c876640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9a6806ea50 0x7f9a6806ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45024/0 (socket says 172.21.15.46:45024) 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6c876640 1 -- 172.21.15.46:0/1416378219 learned_addr learned my addr 172.21.15.46:0/1416378219 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:25.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6d077640 1 -- 172.21.15.46:0/1416378219 >> v1:172.21.15.121:6789/0 conn(0x7f9a68160fb0 legacy=0x7f9a68161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6d077640 1 -- 172.21.15.46:0/1416378219 >> v1:172.21.15.46:6789/0 conn(0x7f9a6806ea50 legacy=0x7f9a6806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a677fe640 1 -- 172.21.15.46:0/1416378219 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414573290 0 0) 0x7f9a68161d20 con 0x7f9a6806ea50 2023-12-16T21:14:25.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.869+0000 7f9a6d077640 1 -- 172.21.15.46:0/1416378219 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6815d9a0 con 0x7f9a68142270 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.870+0000 7f9a6d077640 1 --2- 172.21.15.46:0/1416378219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68142270 0x7f9a681426c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009d90 tx=0x7f9a6002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fdd240e7f96665 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.870+0000 7f9a677fe640 1 -- 172.21.15.46:0/1416378219 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6002cbd0 con 0x7f9a68142270 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.870+0000 7f9a677fe640 1 -- 172.21.15.46:0/1416378219 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6002cd30 con 0x7f9a68142270 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.870+0000 7f9a677fe640 1 -- 172.21.15.46:0/1416378219 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6002cea0 con 0x7f9a68142270 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 -- 172.21.15.46:0/1416378219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68142270 msgr2=0x7f9a681426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 --2- 172.21.15.46:0/1416378219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68142270 0x7f9a681426c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009d90 tx=0x7f9a6002be20 comp rx=0 tx=0).stop 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 -- 172.21.15.46:0/1416378219 shutdown_connections 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 --2- 172.21.15.46:0/1416378219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a68142270 0x7f9a681426c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 -- 172.21.15.46:0/1416378219 >> 172.21.15.46:0/1416378219 conn(0x7f9a6806a6b0 msgr2=0x7f9a6806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 -- 172.21.15.46:0/1416378219 shutdown_connections 2023-12-16T21:14:25.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.871+0000 7f9a6eada640 1 -- 172.21.15.46:0/1416378219 wait complete. 2023-12-16T21:14:25.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.872+0000 7f9a6eada640 1 Processor -- start 2023-12-16T21:14:25.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.872+0000 7f9a6eada640 1 -- start start 2023-12-16T21:14:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.873+0000 7f9a6eada640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 0x7f9a6813cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.873+0000 7f9a6eada640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68142270 0x7f9a68158880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.873+0000 7f9a6eada640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a68160fb0 0x7f9a6813e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.873+0000 7f9a6eada640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68159930 con 0x7f9a6806ea50 2023-12-16T21:14:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.873+0000 7f9a6eada640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68159370 con 0x7f9a68142270 2023-12-16T21:14:25.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.873+0000 7f9a6eada640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a68159650 con 0x7f9a68160fb0 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 0x7f9a6813cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 0x7f9a6813cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52104/0 (socket says 172.21.15.46:52104) 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 -- 172.21.15.46:0/2725364410 learned_addr learned my addr 172.21.15.46:0/2725364410 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6d077640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a68160fb0 0x7f9a6813e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a67fff640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68142270 0x7f9a68158880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 -- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a68160fb0 msgr2=0x7f9a6813e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a68160fb0 0x7f9a6813e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 -- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68142270 msgr2=0x7f9a68158880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68142270 0x7f9a68158880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 -- 172.21.15.46:0/2725364410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60009a70 con 0x7f9a6806ea50 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a67fff640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68142270 0x7f9a68158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6c876640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 0x7f9a6813cdd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00c430 tx=0x7f9a5c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c0074e0 con 0x7f9a6806ea50 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c007660 con 0x7f9a6806ea50 2023-12-16T21:14:25.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6813e090 con 0x7f9a6806ea50 2023-12-16T21:14:25.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c004a20 con 0x7f9a6806ea50 2023-12-16T21:14:25.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.874+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6813d310 con 0x7f9a6806ea50 2023-12-16T21:14:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.875+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6813e5b0 con 0x7f9a6806ea50 2023-12-16T21:14:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.876+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9a5c004b80 con 0x7f9a6806ea50 2023-12-16T21:14:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.876+0000 7f9a657fa640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4c06ae40 0x7f9a4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.876+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9a5c0894c0 con 0x7f9a6806ea50 2023-12-16T21:14:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.876+0000 7f9a67fff640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4c06ae40 0x7f9a4c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:25.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.877+0000 7f9a67fff640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4c06ae40 0x7f9a4c06d300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a6815afe0 tx=0x7f9a54009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:25.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:25.880+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9a6813e5b0 con 0x7f9a6806ea50 2023-12-16T21:14:26.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.056+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a6813e5b0 con 0x7f9a6806ea50 2023-12-16T21:14:26.056 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.057+0000 7f9a657fa640 1 -- 172.21.15.46:0/2725364410 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16754 (secure 0 0 0) 0x7f9a5c054500 con 0x7f9a6806ea50 2023-12-16T21:14:26.056 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:26.057 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:17.654454+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:13:34.042927+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-16T21:13:34.843424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.058+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4c06ae40 msgr2=0x7f9a4c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.058+0000 7f9a6eada640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4c06ae40 0x7f9a4c06d300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a6815afe0 tx=0x7f9a54009040 comp rx=0 tx=0).stop 2023-12-16T21:14:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.058+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 msgr2=0x7f9a6813cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.058+0000 7f9a6eada640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 0x7f9a6813cdd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00c430 tx=0x7f9a5c00c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 shutdown_connections 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9a4c06ae40 0x7f9a4c06d300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a68160fb0 0x7f9a6813e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a68142270 0x7f9a68158880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 --2- 172.21.15.46:0/2725364410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a6806ea50 0x7f9a6813cdd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 >> 172.21.15.46:0/2725364410 conn(0x7f9a6806a6b0 msgr2=0x7f9a6815cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 shutdown_connections 2023-12-16T21:14:26.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.059+0000 7f9a6eada640 1 -- 172.21.15.46:0/2725364410 wait complete. 2023-12-16T21:14:26.069 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-16T21:14:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.177+0000 7efd9c887640 1 Processor -- start 2023-12-16T21:14:26.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.177+0000 7efd9c887640 1 -- start start 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.178+0000 7efd9c887640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980b4370 0x7efd980b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.178+0000 7efd9c887640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd98071770 con 0x7efd980ceb70 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.178+0000 7efd9c887640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd980ccb60 con 0x7efd98144980 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.178+0000 7efd9c887640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9815ced0 con 0x7efd980b4370 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd95d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7efd980ceb70 0x7efd980cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45036/0 (socket says 172.21.15.46:45036) 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd95d9b640 1 -- 172.21.15.46:0/2961072592 learned_addr learned my addr 172.21.15.46:0/2961072592 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd96d9d640 1 --2- 172.21.15.46:0/2961072592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980b4370 0x7efd980b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769196473 0 0) 0x7efd98071770 con 0x7efd980ceb70 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd84003520 con 0x7efd980ceb70 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305562818 0 0) 0x7efd980ccb60 con 0x7efd98144980 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd98071770 con 0x7efd98144980 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd96d9d640 1 -- 172.21.15.46:0/2961072592 >> v1:172.21.15.121:6789/0 conn(0x7efd98144980 legacy=0x7efd9815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd96d9d640 1 -- 172.21.15.46:0/2961072592 >> v1:172.21.15.46:6789/0 conn(0x7efd980ceb70 legacy=0x7efd980cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.179+0000 7efd96d9d640 1 -- 172.21.15.46:0/2961072592 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd980b4ce0 con 0x7efd980b4370 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.180+0000 7efd96d9d640 1 --2- 172.21.15.46:0/2961072592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980b4370 0x7efd980b47a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd980b4e70 tx=0x7efd8802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11cc114991d5efa7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.180+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8802ccd0 con 0x7efd980b4370 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.180+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8802ce30 con 0x7efd980b4370 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.180+0000 7efd9559a640 1 -- 172.21.15.46:0/2961072592 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88034660 con 0x7efd980b4370 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 -- 172.21.15.46:0/2961072592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980b4370 msgr2=0x7efd980b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 --2- 172.21.15.46:0/2961072592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980b4370 0x7efd980b47a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd980b4e70 tx=0x7efd8802bf20 comp rx=0 tx=0).stop 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 -- 172.21.15.46:0/2961072592 shutdown_connections 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 --2- 172.21.15.46:0/2961072592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980b4370 0x7efd980b47a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 -- 172.21.15.46:0/2961072592 >> 172.21.15.46:0/2961072592 conn(0x7efd9806a4d0 msgr2=0x7efd9806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 -- 172.21.15.46:0/2961072592 shutdown_connections 2023-12-16T21:14:26.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.181+0000 7efd9c887640 1 -- 172.21.15.46:0/2961072592 wait complete. 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.182+0000 7efd9c887640 1 Processor -- start 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.182+0000 7efd9c887640 1 -- start start 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9c887640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 0x7efd981f0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9c887640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980ceb70 0x7efd981f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9c887640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd98144980 0x7efd9813ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9c887640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd981f5210 con 0x7efd980b4370 2023-12-16T21:14:26.183 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9c887640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd981f12e0 con 0x7efd98144980 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9c887640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd981f15c0 con 0x7efd980ceb70 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 0x7efd981f0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 0x7efd981f0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52106/0 (socket says 172.21.15.46:52106) 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 -- 172.21.15.46:0/3456125024 learned_addr learned my addr 172.21.15.46:0/3456125024 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd96d9d640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd98144980 0x7efd9813ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd95d9b640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980ceb70 0x7efd981f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 -- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980ceb70 msgr2=0x7efd981f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980ceb70 0x7efd981f4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 -- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd98144980 msgr2=0x7efd9813ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd98144980 0x7efd9813ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 -- 172.21.15.46:0/3456125024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88009a70 con 0x7efd980b4370 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.183+0000 7efd9659c640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 0x7efd981f0cd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c430 tx=0x7efd9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.184+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900052d0 con 0x7efd980b4370 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.184+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90005450 con 0x7efd980b4370 2023-12-16T21:14:26.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.184+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90003830 con 0x7efd980b4370 2023-12-16T21:14:26.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.184+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd981f18a0 con 0x7efd980b4370 2023-12-16T21:14:26.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.184+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd981f1e90 con 0x7efd980b4370 2023-12-16T21:14:26.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.184+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd98069400 con 0x7efd980b4370 2023-12-16T21:14:26.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.185+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7efd90003990 con 0x7efd980b4370 2023-12-16T21:14:26.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.185+0000 7efd8effd640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efd7806ae40 0x7efd7806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.185+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7efd90088cd0 con 0x7efd980b4370 2023-12-16T21:14:26.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.186+0000 7efd95d9b640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efd7806ae40 0x7efd7806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.186+0000 7efd95d9b640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efd7806ae40 0x7efd7806d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efd9815e0a0 tx=0x7efd80007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:26.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.190+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7efd90053d20 con 0x7efd980b4370 2023-12-16T21:14:26.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.404+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efd98140fb0 con 0x7efd980b4370 2023-12-16T21:14:26.404 INFO:tasks.ceph.mon.a.smithi046.stderr:2023-12-16T21:14:26.405+0000 7ff1817f7640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:14:26.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.406+0000 7efd8effd640 1 -- 172.21.15.46:0/3456125024 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7efd90057340 con 0x7efd980b4370 2023-12-16T21:14:26.406 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-16T21:14:26.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efd7806ae40 msgr2=0x7efd7806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efd7806ae40 0x7efd7806d300 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efd9815e0a0 tx=0x7efd80007d30 comp rx=0 tx=0).stop 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 msgr2=0x7efd981f0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 0x7efd981f0cd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c430 tx=0x7efd9000c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 shutdown_connections 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efd7806ae40 0x7efd7806d300 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd98144980 0x7efd9813ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efd980ceb70 0x7efd981f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 --2- 172.21.15.46:0/3456125024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd980b4370 0x7efd981f0cd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.409+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 >> 172.21.15.46:0/3456125024 conn(0x7efd9806a4d0 msgr2=0x7efd9806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:26.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.410+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 shutdown_connections 2023-12-16T21:14:26.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.410+0000 7efd9c887640 1 -- 172.21.15.46:0/3456125024 wait complete. 2023-12-16T21:14:26.410 INFO:tasks.ceph.mds.d.smithi121.stderr: -2871> 2023-12-16T21:13:39.554+0000 7f29fa097640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:14:26.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-16T21:14:26.440 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2023-12-16T21:14:26.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.530+0000 7fb5b091d640 1 Processor -- start 2023-12-16T21:14:26.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.530+0000 7fb5b091d640 1 -- start start 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5b091d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac06ea50 0x7fb5ac06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5b091d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb5ac160fb0 0x7fb5ac161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5b091d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac161d20 con 0x7fb5ac0b3ff0 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5b091d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac0d1ee0 con 0x7fb5ac160fb0 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5b091d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac06f370 con 0x7fb5ac06ea50 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5aad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb5ac0b3ff0 0x7fb5ac0b4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45038/0 (socket says 172.21.15.46:45038) 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5aad9d640 1 -- 172.21.15.46:0/4273256612 learned_addr learned my addr 172.21.15.46:0/4273256612 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/4273256612 >> v2:172.21.15.121:3300/0 conn(0x7fb5ac160fb0 0x7fb5ac161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac06ea50 0x7fb5ac06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/4273256612 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213426465 0 0) 0x7fb5ac161d20 con 0x7fb5ac0b3ff0 2023-12-16T21:14:26.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/4273256612 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598003520 con 0x7fb5ac0b3ff0 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/4273256612 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 624000431 0 0) 0x7fb598003520 con 0x7fb5ac0b3ff0 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a1d9b640 1 -- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac06ea50 msgr2=0x7fb5ac06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac06ea50 0x7fb5ac06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a1d9b640 1 -- 172.21.15.46:0/4273256612 >> v1:172.21.15.46:6789/0 conn(0x7fb5ac0b3ff0 legacy=0x7fb5ac0b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5a1d9b640 1 -- 172.21.15.46:0/4273256612 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac15e760 con 0x7fb5ac160fb0 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.531+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac06ea50 0x7fb5ac06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.532+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac161410 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb5940097a0 tx=0x7fb59402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a6488d5331641f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.532+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/4273256612 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40019f0 con 0x7fb5ac160fb0 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.532+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/4273256612 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59402e050 con 0x7fb5ac160fb0 2023-12-16T21:14:26.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5a9d9b640 1 -- 172.21.15.46:0/4273256612 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb594030730 con 0x7fb5ac160fb0 2023-12-16T21:14:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 -- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 msgr2=0x7fb5ac161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac161410 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb5940097a0 tx=0x7fb59402b710 comp rx=0 tx=0).stop 2023-12-16T21:14:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 -- 172.21.15.46:0/4273256612 shutdown_connections 2023-12-16T21:14:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac161410 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 --2- 172.21.15.46:0/4273256612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac06ea50 0x7fb5ac06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 -- 172.21.15.46:0/4273256612 >> 172.21.15.46:0/4273256612 conn(0x7fb5ac06a6b0 msgr2=0x7fb5ac06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 -- 172.21.15.46:0/4273256612 shutdown_connections 2023-12-16T21:14:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.533+0000 7fb5b091d640 1 -- 172.21.15.46:0/4273256612 wait complete. 2023-12-16T21:14:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.534+0000 7fb5b091d640 1 Processor -- start 2023-12-16T21:14:26.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.534+0000 7fb5b091d640 1 -- start start 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5b091d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 0x7fb5ac156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5b091d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac0b3ff0 0x7fb5ac06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5b091d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac0b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5b091d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac157170 con 0x7fb5ac06ea50 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5b091d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac13a410 con 0x7fb5ac160fb0 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5b091d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac06b5e0 con 0x7fb5ac0b3ff0 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5aa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 0x7fb5ac156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5aa59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 0x7fb5ac156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52116/0 (socket says 172.21.15.46:52116) 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5aa59c640 1 -- 172.21.15.46:0/1858647371 learned_addr learned my addr 172.21.15.46:0/1858647371 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5aad9d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac0b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.535+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac0b3ff0 0x7fb5ac06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5aa59c640 1 -- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac0b3ff0 msgr2=0x7fb5ac06c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac0b3ff0 0x7fb5ac06c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5aa59c640 1 -- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 msgr2=0x7fb5ac0b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac0b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5aa59c640 1 -- 172.21.15.46:0/1858647371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c009400 con 0x7fb5ac06ea50 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5aa59c640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 0x7fb5ac156810 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00e960 tx=0x7fb59c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c00ca00 con 0x7fb5ac06ea50 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c00cb60 con 0x7fb5ac06ea50 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c013470 con 0x7fb5ac06ea50 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb594009400 con 0x7fb5ac06ea50 2023-12-16T21:14:26.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.536+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac06bba0 con 0x7fb5ac06ea50 2023-12-16T21:14:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.537+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac069400 con 0x7fb5ac06ea50 2023-12-16T21:14:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.538+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb59c0135d0 con 0x7fb5ac06ea50 2023-12-16T21:14:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.538+0000 7fb5a37fe640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb58c06ae40 0x7fb58c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.538+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fb59c089820 con 0x7fb5ac06ea50 2023-12-16T21:14:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.538+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb58c06ae40 0x7fb58c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:26.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.539+0000 7fb5a1d9b640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb58c06ae40 0x7fb58c06d300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb594034000 tx=0x7fb594008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:26.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.542+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb59c054940 con 0x7fb5ac06ea50 2023-12-16T21:14:26.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:26.757+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb5ac070d30 con 0x7fb5ac06ea50 2023-12-16T21:14:27.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:14:27.325 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:14:27.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.408+0000 7fb5a37fe640 1 -- 172.21.15.46:0/1858647371 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb59c057f60 con 0x7fb5ac06ea50 2023-12-16T21:14:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb58c06ae40 msgr2=0x7fb58c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:27.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb58c06ae40 0x7fb58c06d300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb594034000 tx=0x7fb594008030 comp rx=0 tx=0).stop 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 msgr2=0x7fb5ac156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 0x7fb5ac156810 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00e960 tx=0x7fb59c00ee20 comp rx=0 tx=0).stop 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 shutdown_connections 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb58c06ae40 0x7fb58c06d300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5ac160fb0 0x7fb5ac0b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5ac0b3ff0 0x7fb5ac06c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 --2- 172.21.15.46:0/1858647371 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5ac06ea50 0x7fb5ac156810 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 >> 172.21.15.46:0/1858647371 conn(0x7fb5ac06a6b0 msgr2=0x7fb5ac15c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:27.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 shutdown_connections 2023-12-16T21:14:27.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.411+0000 7fb5b091d640 1 -- 172.21.15.46:0/1858647371 wait complete. 2023-12-16T21:14:27.423 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-16T21:14:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.578+0000 7f1c1e773640 1 Processor -- start 2023-12-16T21:14:27.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.578+0000 7f1c1e773640 1 -- start start 2023-12-16T21:14:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c1e773640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c1815d330 0x7f1c180b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c1e773640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c180b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:27.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c1e773640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c180b2b10 con 0x7f1c180d2910 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c1e773640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c180d1590 con 0x7f1c1815d330 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c1e773640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180d1870 con 0x7f1c180d3900 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c17fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c1815d330 0x7f1c180b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c17fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1c1815d330 0x7f1c180b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52122/0 (socket says 172.21.15.46:52122) 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.579+0000 7f1c17fff640 1 -- 172.21.15.46:0/3137061329 learned_addr learned my addr 172.21.15.46:0/3137061329 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.580+0000 7f1c1cd10640 1 --2- 172.21.15.46:0/3137061329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c180b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.580+0000 7f1c17fff640 1 -- 172.21.15.46:0/3137061329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 msgr2=0x7f1c180b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.580+0000 7f1c17fff640 1 --2- 172.21.15.46:0/3137061329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c180b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.580+0000 7f1c17fff640 1 -- 172.21.15.46:0/3137061329 >> v1:172.21.15.121:6789/0 conn(0x7f1c180d2910 legacy=0x7f1c180d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:27.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.580+0000 7f1c17fff640 1 -- 172.21.15.46:0/3137061329 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c180d1b50 con 0x7f1c1815d330 2023-12-16T21:14:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.580+0000 7f1c17fff640 1 --2- 172.21.15.46:0/3137061329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1815d330 0x7f1c180b8b80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009870 tx=0x7f1c0802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839dbee8c44c7b41 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c16ffd640 1 -- 172.21.15.46:0/3137061329 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080317c0 con 0x7f1c1815d330 2023-12-16T21:14:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c16ffd640 1 -- 172.21.15.46:0/3137061329 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c08031920 con 0x7f1c1815d330 2023-12-16T21:14:27.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c16ffd640 1 -- 172.21.15.46:0/3137061329 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08031cb0 con 0x7f1c1815d330 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 -- 172.21.15.46:0/3137061329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1815d330 msgr2=0x7f1c180b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 --2- 172.21.15.46:0/3137061329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1815d330 0x7f1c180b8b80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009870 tx=0x7f1c0802b860 comp rx=0 tx=0).stop 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 -- 172.21.15.46:0/3137061329 shutdown_connections 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 --2- 172.21.15.46:0/3137061329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c180b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 --2- 172.21.15.46:0/3137061329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c1815d330 0x7f1c180b8b80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 -- 172.21.15.46:0/3137061329 >> 172.21.15.46:0/3137061329 conn(0x7f1c18072020 msgr2=0x7f1c18072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 -- 172.21.15.46:0/3137061329 shutdown_connections 2023-12-16T21:14:27.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.581+0000 7f1c1e773640 1 -- 172.21.15.46:0/3137061329 wait complete. 2023-12-16T21:14:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.582+0000 7f1c1e773640 1 Processor -- start 2023-12-16T21:14:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.582+0000 7f1c1e773640 1 -- start start 2023-12-16T21:14:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1e773640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 0x7f1c181430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1e773640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c18160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1e773640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c1815d330 0x7f1c181560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:27.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1e773640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180c86c0 con 0x7f1c180d2910 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1e773640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c18156b60 con 0x7f1c1815d330 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1e773640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c180c7d20 con 0x7f1c180d3900 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 0x7f1c181430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c17fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 0x7f1c181430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52138/0 (socket says 172.21.15.46:52138) 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c17fff640 1 -- 172.21.15.46:0/4199161355 learned_addr learned my addr 172.21.15.46:0/4199161355 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c1cd10640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c1815d330 0x7f1c181560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.583+0000 7f1c177fe640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c18160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c17fff640 1 -- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 msgr2=0x7f1c18160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c17fff640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c18160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c17fff640 1 -- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c1815d330 msgr2=0x7f1c181560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c17fff640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c1815d330 0x7f1c181560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c17fff640 1 -- 172.21.15.46:0/4199161355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080094d0 con 0x7f1c180d2910 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c177fe640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c18160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:27.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c17fff640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 0x7f1c181430d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1c08000c00 tx=0x7f1c080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08032860 con 0x7f1c180d2910 2023-12-16T21:14:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c080329c0 con 0x7f1c180d2910 2023-12-16T21:14:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c180c8000 con 0x7f1c180d2910 2023-12-16T21:14:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08032d70 con 0x7f1c180d2910 2023-12-16T21:14:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.584+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c18145f20 con 0x7f1c180d2910 2023-12-16T21:14:27.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.585+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c180d2da0 con 0x7f1c180d2910 2023-12-16T21:14:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.586+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1c0804e450 con 0x7f1c180d2910 2023-12-16T21:14:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.586+0000 7f1c14ff9640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c0406ae40 0x7f1c0406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.586+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f1c080ab790 con 0x7f1c180d2910 2023-12-16T21:14:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.586+0000 7f1c177fe640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c0406ae40 0x7f1c0406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.587+0000 7f1c177fe640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c0406ae40 0x7f1c0406d300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c006fd0 tx=0x7f1c0c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:27.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.590+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1c08076800 con 0x7f1c180d2910 2023-12-16T21:14:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:27.766+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1c18069400 con 0x7f1c180d2910 2023-12-16T21:14:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.694+0000 7f1c14ff9640 1 -- 172.21.15.46:0/4199161355 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1c08079e20 con 0x7f1c180d2910 2023-12-16T21:14:28.694 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c0406ae40 msgr2=0x7f1c0406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c0406ae40 0x7f1c0406d300 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c006fd0 tx=0x7f1c0c008040 comp rx=0 tx=0).stop 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 msgr2=0x7f1c181430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 0x7f1c181430d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1c08000c00 tx=0x7f1c080046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 shutdown_connections 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1c0406ae40 0x7f1c0406d300 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c1815d330 0x7f1c181560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1c180d3900 0x7f1c18160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 --2- 172.21.15.46:0/4199161355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c180d2910 0x7f1c181430d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 >> 172.21.15.46:0/4199161355 conn(0x7f1c18072020 msgr2=0x7f1c18072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:28.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 shutdown_connections 2023-12-16T21:14:28.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.698+0000 7f1c1e773640 1 -- 172.21.15.46:0/4199161355 wait complete. 2023-12-16T21:14:28.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-16T21:14:28.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.830+0000 7f1a0c644640 1 Processor -- start 2023-12-16T21:14:28.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.830+0000 7f1a0c644640 1 -- start start 2023-12-16T21:14:28.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.831+0000 7f1a0c644640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a040d2910 0x7f1a040d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:28.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.831+0000 7f1a0c644640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a040d3730 0x7f1a040b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:28.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.831+0000 7f1a0c644640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a040b2b10 con 0x7f1a0415d330 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.831+0000 7f1a0c644640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a040d1590 con 0x7f1a040d2910 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.831+0000 7f1a0c644640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a040d1870 con 0x7f1a040d3730 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.831+0000 7f1a09bdf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1a040d2910 0x7f1a040d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a0a3e0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1a0415d330 0x7f1a040b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45048/0 (socket says 172.21.15.46:45048) 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a0a3e0640 1 -- 172.21.15.46:0/151323561 learned_addr learned my addr 172.21.15.46:0/151323561 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a0abe1640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a040d3730 0x7f1a040b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a09bdf640 1 -- 172.21.15.46:0/151323561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a040d3730 msgr2=0x7f1a040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a09bdf640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a040d3730 0x7f1a040b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a09bdf640 1 -- 172.21.15.46:0/151323561 >> v1:172.21.15.46:6789/0 conn(0x7f1a0415d330 legacy=0x7f1a040b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a09bdf640 1 -- 172.21.15.46:0/151323561 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a040d3b90 con 0x7f1a040d2910 2023-12-16T21:14:28.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.832+0000 7f1a0abe1640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a040d3730 0x7f1a040b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:28.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a09bdf640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d2910 0x7f1a040d31f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1a000097a0 tx=0x7f1a0002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74bc8b0439d1dba server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:28.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a093de640 1 -- 172.21.15.46:0/151323561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0002e070 con 0x7f1a040d2910 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a093de640 1 -- 172.21.15.46:0/151323561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a000315a0 con 0x7f1a040d2910 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a093de640 1 -- 172.21.15.46:0/151323561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00031930 con 0x7f1a040d2910 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a0c644640 1 -- 172.21.15.46:0/151323561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d2910 msgr2=0x7f1a040d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a0c644640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d2910 0x7f1a040d31f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1a000097a0 tx=0x7f1a0002b840 comp rx=0 tx=0).stop 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a0c644640 1 -- 172.21.15.46:0/151323561 shutdown_connections 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a0c644640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a040d3730 0x7f1a040b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a0c644640 1 --2- 172.21.15.46:0/151323561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d2910 0x7f1a040d31f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.833+0000 7f1a0c644640 1 -- 172.21.15.46:0/151323561 >> 172.21.15.46:0/151323561 conn(0x7f1a04072020 msgr2=0x7f1a04072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.834+0000 7f1a0c644640 1 -- 172.21.15.46:0/151323561 shutdown_connections 2023-12-16T21:14:28.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.834+0000 7f1a0c644640 1 -- 172.21.15.46:0/151323561 wait complete. 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 Processor -- start 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 -- start start 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a040d2910 0x7f1a0414fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 0x7f1a0414c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a0415d330 0x7f1a0414cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0415f200 con 0x7f1a040d3730 2023-12-16T21:14:28.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0415f4a0 con 0x7f1a040d2910 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.835+0000 7f1a0c644640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0415dd60 con 0x7f1a0415d330 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 0x7f1a0414c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 0x7f1a0414c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52164/0 (socket says 172.21.15.46:52164) 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 -- 172.21.15.46:0/1141329225 learned_addr learned my addr 172.21.15.46:0/1141329225 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a0a3e0640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a040d2910 0x7f1a0414fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a0abe1640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a0415d330 0x7f1a0414cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 -- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a0415d330 msgr2=0x7f1a0414cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a0415d330 0x7f1a0414cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 -- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a040d2910 msgr2=0x7f1a0414fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a040d2910 0x7f1a0414fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 -- 172.21.15.46:0/1141329225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a00009400 con 0x7f1a040d3730 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a09bdf640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 0x7f1a0414c540 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1a00034040 tx=0x7f1a000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00032840 con 0x7f1a040d3730 2023-12-16T21:14:28.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a000329a0 con 0x7f1a040d3730 2023-12-16T21:14:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0415da80 con 0x7f1a040d3730 2023-12-16T21:14:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00032d50 con 0x7f1a040d3730 2023-12-16T21:14:28.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.836+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0415fa20 con 0x7f1a040d3730 2023-12-16T21:14:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.837+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0415fc10 con 0x7f1a040d3730 2023-12-16T21:14:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.838+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1a0004e430 con 0x7f1a040d3730 2023-12-16T21:14:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.838+0000 7f19f6ffd640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19e806ae40 0x7f19e806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.838+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f1a000ab500 con 0x7f1a040d3730 2023-12-16T21:14:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.838+0000 7f1a0a3e0640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19e806ae40 0x7f19e806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:28.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.839+0000 7f1a0a3e0640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19e806ae40 0x7f19e806d300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f19f8005880 tx=0x7f19f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:28.842 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:28.842+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1a000766a0 con 0x7f1a040d3730 2023-12-16T21:14:29.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.018+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1a04159f60 con 0x7f1a040d3730 2023-12-16T21:14:29.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.696+0000 7f19f6ffd640 1 -- 172.21.15.46:0/1141329225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1a00079cc0 con 0x7f1a040d3730 2023-12-16T21:14:29.696 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2023-12-16T21:14:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19e806ae40 msgr2=0x7f19e806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19e806ae40 0x7f19e806d300 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f19f8005880 tx=0x7f19f8009040 comp rx=0 tx=0).stop 2023-12-16T21:14:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 msgr2=0x7f1a0414c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 0x7f1a0414c540 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1a00034040 tx=0x7f1a000046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 shutdown_connections 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f19e806ae40 0x7f19e806d300 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1a040d3730 0x7f1a0414c540 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a0415d330 0x7f1a0414cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 --2- 172.21.15.46:0/1141329225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a040d2910 0x7f1a0414fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.700+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 >> 172.21.15.46:0/1141329225 conn(0x7f1a04072020 msgr2=0x7f1a04072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.701+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 shutdown_connections 2023-12-16T21:14:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.701+0000 7f1a0c644640 1 -- 172.21.15.46:0/1141329225 wait complete. 2023-12-16T21:14:29.717 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-16T21:14:29.717 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-16T21:14:29.717 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-16T21:14:29.717 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:29.718 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-16T21:14:29.718 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-16T21:14:29.718 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-16T21:14:29.718 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:29.718 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-16T21:14:29.718 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-16T21:14:29.718 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-16T21:14:29.719 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:29.719 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-16T21:14:29.719 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-16T21:14:29.719 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-16T21:14:29.719 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:29.729 INFO:tasks.ceph.mds.b:Stopped 2023-12-16T21:14:29.732 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-16T21:14:29.733 INFO:tasks.ceph.mds.d:Stopped 2023-12-16T21:14:29.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-16T21:14:29.734 INFO:tasks.ceph.mds.a:Stopped 2023-12-16T21:14:29.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-16T21:14:29.735 INFO:tasks.ceph.mds.c:Stopped 2023-12-16T21:14:29.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-16T21:14:29.771 INFO:tasks.ceph.mds.b:Started 2023-12-16T21:14:29.772 DEBUG:teuthology.parallel:result is None 2023-12-16T21:14:29.772 INFO:tasks.ceph.mds.d:Started 2023-12-16T21:14:29.772 DEBUG:teuthology.parallel:result is None 2023-12-16T21:14:29.777 INFO:tasks.ceph.mds.a:Started 2023-12-16T21:14:29.778 DEBUG:teuthology.parallel:result is None 2023-12-16T21:14:29.778 INFO:tasks.ceph.mds.c:Started 2023-12-16T21:14:29.779 DEBUG:teuthology.parallel:result is None 2023-12-16T21:14:29.779 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-16T21:14:29.851 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2023-12-16T21:14:29.852 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-12-16T21:14:29.852 INFO:tasks.ceph.mds.b.smithi121.stderr:2023-12-16T21:14:29.852+0000 7f88a9f3c640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2023-12-16T21:14:29.866 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-12-16T21:14:29.866 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-12-16T21:14:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.898+0000 7ff46c82d640 1 Processor -- start 2023-12-16T21:14:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.898+0000 7ff46c82d640 1 -- start start 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46c82d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4680d3900 0x7ff4680b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46c82d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4680b2b10 con 0x7ff46815d330 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff46815d330 0x7ff4680b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:45062/0 (socket says 172.21.15.46:45062) 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46659c640 1 -- 172.21.15.46:0/1141539302 learned_addr learned my addr 172.21.15.46:0/1141539302 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4680d1590 con 0x7ff4680d2910 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff466d9d640 1 --2- 172.21.15.46:0/1141539302 >> v2:172.21.15.46:3300/0 conn(0x7ff4680d3900 0x7ff4680b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4680d1870 con 0x7ff4680d3900 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46559a640 1 -- 172.21.15.46:0/1141539302 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264938198 0 0) 0x7ff4680b2b10 con 0x7ff46815d330 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff46559a640 1 -- 172.21.15.46:0/1141539302 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff450003520 con 0x7ff46815d330 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff466d9d640 1 -- 172.21.15.46:0/1141539302 >> v1:172.21.15.121:6789/0 conn(0x7ff4680d2910 legacy=0x7ff4680d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff466d9d640 1 -- 172.21.15.46:0/1141539302 >> v1:172.21.15.46:6789/0 conn(0x7ff46815d330 legacy=0x7ff4680b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.899+0000 7ff466d9d640 1 -- 172.21.15.46:0/1141539302 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4680d1b50 con 0x7ff4680d3900 2023-12-16T21:14:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.900+0000 7ff466d9d640 1 --2- 172.21.15.46:0/1141539302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d3900 0x7ff4680b6390 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff460009d90 tx=0x7ff46002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555d2b58da1e70e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.900+0000 7ff46559a640 1 -- 172.21.15.46:0/1141539302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46002cc60 con 0x7ff4680d3900 2023-12-16T21:14:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.900+0000 7ff46559a640 1 -- 172.21.15.46:0/1141539302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46002cdc0 con 0x7ff4680d3900 2023-12-16T21:14:29.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.900+0000 7ff46559a640 1 -- 172.21.15.46:0/1141539302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460034660 con 0x7ff4680d3900 2023-12-16T21:14:29.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d3900 msgr2=0x7ff4680b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 --2- 172.21.15.46:0/1141539302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d3900 0x7ff4680b6390 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff460009d90 tx=0x7ff46002beb0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 shutdown_connections 2023-12-16T21:14:29.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 --2- 172.21.15.46:0/1141539302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d3900 0x7ff4680b6390 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 >> 172.21.15.46:0/1141539302 conn(0x7ff468072020 msgr2=0x7ff468072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:29.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 shutdown_connections 2023-12-16T21:14:29.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.901+0000 7ff46c82d640 1 -- 172.21.15.46:0/1141539302 wait complete. 2023-12-16T21:14:29.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.902+0000 7ff46c82d640 1 Processor -- start 2023-12-16T21:14:29.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 -- start start 2023-12-16T21:14:29.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 0x7ff4680cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:29.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4680d3900 0x7ff4681f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:29.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff46815d330 0x7ff4681f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680cc3c0 con 0x7ff4680d2910 2023-12-16T21:14:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680ccd20 con 0x7ff4680d3900 2023-12-16T21:14:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46c82d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4680cb740 con 0x7ff46815d330 2023-12-16T21:14:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 0x7ff4680cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 0x7ff4680cc7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52200/0 (socket says 172.21.15.46:52200) 2023-12-16T21:14:29.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.903+0000 7ff46659c640 1 -- 172.21.15.46:0/709524346 learned_addr learned my addr 172.21.15.46:0/709524346 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff466d9d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff46815d330 0x7ff4681f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff465d9b640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4680d3900 0x7ff4681f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46659c640 1 -- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff46815d330 msgr2=0x7ff4681f67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46659c640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff46815d330 0x7ff4681f67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46659c640 1 -- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4680d3900 msgr2=0x7ff4681f6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46659c640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4680d3900 0x7ff4681f6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:29.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46659c640 1 -- 172.21.15.46:0/709524346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff460009a70 con 0x7ff4680d2910 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff465d9b640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4680d3900 0x7ff4681f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff466d9d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff46815d330 0x7ff4681f67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46659c640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 0x7ff4680cc7e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff45800c410 tx=0x7ff45800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458005ce0 con 0x7ff4680d2910 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff458005e40 con 0x7ff4680d2910 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4580072f0 con 0x7ff4680d2910 2023-12-16T21:14:29.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4680cba20 con 0x7ff4680d2910 2023-12-16T21:14:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.904+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4680cbfe0 con 0x7ff4680d2910 2023-12-16T21:14:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.905+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4680cc500 con 0x7ff4680d2910 2023-12-16T21:14:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.905+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff458007510 con 0x7ff4680d2910 2023-12-16T21:14:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.906+0000 7ff456ffd640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff44806ae40 0x7ff44806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:29.912 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.906+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff458088970 con 0x7ff4680d2910 2023-12-16T21:14:29.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.906+0000 7ff465d9b640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff44806ae40 0x7ff44806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:29.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.907+0000 7ff465d9b640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff44806ae40 0x7ff44806d300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff4680b48a0 tx=0x7ff45c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:29.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:29.910+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff4680cc500 con 0x7ff4680d2910 2023-12-16T21:14:30.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.099+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff468069400 con 0x7ff4680d2910 2023-12-16T21:14:30.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.718+0000 7ff456ffd640 1 -- 172.21.15.46:0/709524346 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v28) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff458017080 con 0x7ff4680d2910 2023-12-16T21:14:30.718 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2023-12-16T21:14:30.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff44806ae40 msgr2=0x7ff44806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:30.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff44806ae40 0x7ff44806d300 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff4680b48a0 tx=0x7ff45c009040 comp rx=0 tx=0).stop 2023-12-16T21:14:30.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 msgr2=0x7ff4680cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:30.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 0x7ff4680cc7e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff45800c410 tx=0x7ff45800c8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 shutdown_connections 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff44806ae40 0x7ff44806d300 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff46815d330 0x7ff4681f67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4680d3900 0x7ff4681f6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 --2- 172.21.15.46:0/709524346 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4680d2910 0x7ff4680cc7e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 >> 172.21.15.46:0/709524346 conn(0x7ff468072020 msgr2=0x7ff468072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 shutdown_connections 2023-12-16T21:14:30.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.720+0000 7ff46c82d640 1 -- 172.21.15.46:0/709524346 wait complete. 2023-12-16T21:14:30.736 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-16T21:14:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.848+0000 7fda3a4d0640 1 Processor -- start 2023-12-16T21:14:30.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.848+0000 7fda3a4d0640 1 -- start start 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda3a4d0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fda3415d330 0x7fda340b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda3a4d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda340b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda3a4d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda340b2b10 con 0x7fda340d2910 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda3a4d0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda340d1590 con 0x7fda3415d330 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda3a4d0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda340d1870 con 0x7fda340d3900 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda337fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fda340d2910 0x7fda340d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36488/0 (socket says 172.21.15.46:36488) 2023-12-16T21:14:30.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.849+0000 7fda337fe640 1 -- 172.21.15.46:0/3444545347 learned_addr learned my addr 172.21.15.46:0/3444545347 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda33fff640 1 --2- 172.21.15.46:0/3444545347 >> v2:172.21.15.121:3300/0 conn(0x7fda3415d330 0x7fda340b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda38a6d640 1 --2- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda340b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda32ffd640 1 -- 172.21.15.46:0/3444545347 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192534465 0 0) 0x7fda340b2b10 con 0x7fda340d2910 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda32ffd640 1 -- 172.21.15.46:0/3444545347 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda1c003520 con 0x7fda340d2910 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda33fff640 1 -- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 msgr2=0x7fda340b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda33fff640 1 --2- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda340b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda33fff640 1 -- 172.21.15.46:0/3444545347 >> v1:172.21.15.46:6789/0 conn(0x7fda340d2910 legacy=0x7fda340d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda32ffd640 1 -- 172.21.15.46:0/3444545347 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458969389 0 0) 0x7fda1c003520 con 0x7fda340d2910 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.850+0000 7fda33fff640 1 -- 172.21.15.46:0/3444545347 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda340d1b50 con 0x7fda3415d330 2023-12-16T21:14:30.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.851+0000 7fda33fff640 1 --2- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340b8b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda240097a0 tx=0x7fda2402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8f8b95b1ec2e669 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.851+0000 7fda32ffd640 1 -- 172.21.15.46:0/3444545347 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28001670 con 0x7fda3415d330 2023-12-16T21:14:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.851+0000 7fda32ffd640 1 -- 172.21.15.46:0/3444545347 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2402e050 con 0x7fda3415d330 2023-12-16T21:14:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.851+0000 7fda32ffd640 1 -- 172.21.15.46:0/3444545347 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24030710 con 0x7fda3415d330 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 msgr2=0x7fda340b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340b8b80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda240097a0 tx=0x7fda2402b710 comp rx=0 tx=0).stop 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3444545347 shutdown_connections 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda340b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3444545347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340b8b80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3444545347 >> 172.21.15.46:0/3444545347 conn(0x7fda34072020 msgr2=0x7fda34072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3444545347 shutdown_connections 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.852+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3444545347 wait complete. 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.853+0000 7fda3a4d0640 1 Processor -- start 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.853+0000 7fda3a4d0640 1 -- start start 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda3a4d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 0x7fda3414a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda3a4d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda3415d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda3a4d0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340c9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:30.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda3a4d0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda340c82b0 con 0x7fda340d2910 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda3a4d0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda340c8590 con 0x7fda3415d330 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda3a4d0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda340c8870 con 0x7fda340d3900 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 0x7fda3414a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 0x7fda3414a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41432/0 (socket says 172.21.15.46:41432) 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 -- 172.21.15.46:0/3534011569 learned_addr learned my addr 172.21.15.46:0/3534011569 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda337fe640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda3415d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda38a6d640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 -- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 msgr2=0x7fda3415d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda3415d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 -- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 msgr2=0x7fda340c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 -- 172.21.15.46:0/3534011569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24009400 con 0x7fda340d2910 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda337fe640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda3415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda38a6d640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda33fff640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 0x7fda3414a5e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fda24000c00 tx=0x7fda240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2403c620 con 0x7fda340d2910 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.854+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2403c780 con 0x7fda340d2910 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.855+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda340c8b50 con 0x7fda340d2910 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.855+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2403cb30 con 0x7fda340d2910 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.855+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3414ab20 con 0x7fda340d2910 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.856+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda340c9070 con 0x7fda340d2910 2023-12-16T21:14:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.856+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fda24039070 con 0x7fda340d2910 2023-12-16T21:14:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.856+0000 7fda30ff9640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.856+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda240aaad0 con 0x7fda340d2910 2023-12-16T21:14:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.857+0000 7fda337fe640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.857+0000 7fda337fe640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fda3415e490 tx=0x7fda28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:30.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:30.861+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fda24075e50 con 0x7fda340d2910 2023-12-16T21:14:31.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.102+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fda34069400 con 0x7fda340d2910 2023-12-16T21:14:31.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.103+0000 7fda30ff9640 1 -- 172.21.15.46:0/3534011569 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v8) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fda24079470 con 0x7fda340d2910 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lilFjJRAAS3jGgFJPZWkcXnfWFuSkmw==", 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:14:31.103 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5liiXVMhAAJWJpsjXpIJHc/8ySyArSgQ==", 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.104 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lElZsLRAA5EGpXLzgGQdhS3J+Pf+9gw==", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5l7uY3OhAAwqyF6WkAH8BGy1tWJygLCg==", 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.105 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC/En5lteX6JxAAOUBR8wDo1UbdDenhHLeSNg==", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.106 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDCEn5le0WlJxAAeyeKmj4yjwDCNn0WuoW4Rw==", 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.107 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDFEn5l/ZafJhAA72K0MtXsz0kQAa0Hhlrg5g==", 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.108 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDIEn5l/BX4JRAAcfb0TNdwljS3Hwp6E3EPXw==", 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.109 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDOEn5lmZNQLRAAZ4THlfKtWJxeQXvoVShsRg==", 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.110 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDREn5lPZlmLBAABoR8+M5fdPUah5hZITIABA==", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDUEn5lkOKRKxAAa5eJiAzXy/L79xL7egwC+g==", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.111 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDXEn5lyQ1RKhAAcFWeiQsS4BaAJ/fIgCiCPA==", 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2023-12-16T21:14:31.112 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w==", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5l7si2ChAAPixHXNSDvbDoEOOjVYonbQ==", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:14:31.113 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5l66kFBxAApn9h5R4KlGm9Udlz7FGldw==", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2023-12-16T21:14:31.114 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5l1NSAEBAAqlxrkimPL3Z1z3llw4xr9Q==", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lgH3XBhAA9d1n5heC1V63dyfuAmerGA==", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:14:31.115 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mds", 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDnEn5lf/wILBAAQatPaF4KXU11Ag3ToGy9hA==", 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mds" 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-mgr", 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDnEn5lZwoJLBAATG2dD5tBorucQtGCXUmGow==", 2023-12-16T21:14:31.116 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-mgr" 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-osd", 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDnEn5lehMJLBAAjwhwcIiw09/yJiBGO2wMog==", 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-osd" 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd", 2023-12-16T21:14:31.117 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDnEn5lKB8JLBAAJZdWTbmbjiFUyAhPxzM06A==", 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd" 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDnEn5l+CYJLBAACkdzl6z2nMG9nGQP8kun9Q==", 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.bootstrap-rgw", 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDnEn5lMC8JLBAA1rtM12zHNbEOWppJZZQpVw==", 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.118 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile bootstrap-rgw" 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lD2F+IRAAD5Xq3cx5Gd9fKtnuYe9HMw==", 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2023-12-16T21:14:31.119 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lgvZiHRAA87oDIk7xwjmYe4gMMg1NTQ==", 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2023-12-16T21:14:31.120 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 msgr2=0x7fda2006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fda3415e490 tx=0x7fda28009040 comp rx=0 tx=0).stop 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 msgr2=0x7fda3414a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 0x7fda3414a5e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fda24000c00 tx=0x7fda240046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 shutdown_connections 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3415d330 0x7fda340c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda340d3900 0x7fda3415d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 --2- 172.21.15.46:0/3534011569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda340d2910 0x7fda3414a5e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 >> 172.21.15.46:0/3534011569 conn(0x7fda34072020 msgr2=0x7fda340b6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:31.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 shutdown_connections 2023-12-16T21:14:31.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.105+0000 7fda3a4d0640 1 -- 172.21.15.46:0/3534011569 wait complete. 2023-12-16T21:14:31.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-12-16T21:14:31.230 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.231+0000 7fc1f72b4640 1 Processor -- start 2023-12-16T21:14:31.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.231+0000 7fc1f72b4640 1 -- start start 2023-12-16T21:14:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f72b4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 0x7fc1f006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f72b4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc1f00ad640 0x7fc1f006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f72b4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f00adae0 con 0x7fc1f00aeb80 2023-12-16T21:14:31.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f72b4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f00addc0 con 0x7fc1f00ad640 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f72b4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00ae0a0 con 0x7fc1f015d1d0 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f5851640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc1f00aeb80 0x7fc1f0160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36496/0 (socket says 172.21.15.46:36496) 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f5851640 1 -- 172.21.15.46:0/1107739515 learned_addr learned my addr 172.21.15.46:0/1107739515 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1107739515 >> v2:172.21.15.121:3300/0 conn(0x7fc1f00ad640 0x7fc1f006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1f5050640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 0x7fc1f006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922076200 0 0) 0x7fc1f00adae0 con 0x7fc1f00aeb80 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1dc003520 con 0x7fc1f00aeb80 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 551200556 0 0) 0x7fc1dc003520 con 0x7fc1f00aeb80 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 msgr2=0x7fc1f006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 0x7fc1f006b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 >> v2:172.21.15.121:3300/0 conn(0x7fc1f00ad640 msgr2=0x7fc1f006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 --2- 172.21.15.46:0/1107739515 >> v2:172.21.15.121:3300/0 conn(0x7fc1f00ad640 0x7fc1f006ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.234 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f00ae380 con 0x7fc1f00aeb80 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fc1e80019f0 con 0x7fc1f00aeb80 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.232+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 >> v1:172.21.15.46:6789/0 conn(0x7fc1f00aeb80 legacy=0x7fc1f0160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1e7fff640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 0x7fc1dc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1e7fff640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1dc003ef0 0x7fc1dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1e7fff640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc004d00 0x7fc1dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00addc0 con 0x7fc1dc003ef0 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00ae0a0 con 0x7fc1dc004d00 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1dc00b980 con 0x7fc1f015d1d0 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1dc003ef0 0x7fc1dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f5050640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 0x7fc1dc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f5851640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc004d00 0x7fc1dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f484f640 1 -- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 msgr2=0x7fc1dc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f015d1d0 0x7fc1dc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f484f640 1 -- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc004d00 msgr2=0x7fc1dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:31.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc004d00 0x7fc1dc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.233+0000 7fc1f484f640 1 -- 172.21.15.46:0/1107739515 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc003520 con 0x7fc1dc003ef0 2023-12-16T21:14:31.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1dc003ef0 0x7fc1dc0047c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009ac0 tx=0x7fc1dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33594c4589d8548f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e002c6d0 con 0x7fc1dc003ef0 2023-12-16T21:14:31.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e002c830 con 0x7fc1dc003ef0 2023-12-16T21:14:31.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1e7fff640 1 -- 172.21.15.46:0/1107739515 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e002cbc0 con 0x7fc1dc003ef0 2023-12-16T21:14:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1107739515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1dc003ef0 msgr2=0x7fc1dc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1107739515 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1dc003ef0 0x7fc1dc0047c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009ac0 tx=0x7fc1dc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f5851640 1 -- 172.21.15.46:0/1107739515 reap_dead start 2023-12-16T21:14:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1107739515 shutdown_connections 2023-12-16T21:14:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1107739515 >> 172.21.15.46:0/1107739515 conn(0x7fc1f0070e70 msgr2=0x7fc1f0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:31.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1107739515 shutdown_connections 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.234+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1107739515 wait complete. 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 Processor -- start 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 -- start start 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc003ef0 0x7fc1f0146870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 0x7fc1f00af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f0147140 0x7fc1f01475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f00af970 con 0x7fc1f00aefd0 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0148ad0 con 0x7fc1dc003ef0 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f72b4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f0146db0 con 0x7fc1f0147140 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f484f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 0x7fc1f00af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f484f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 0x7fc1f00af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41464/0 (socket says 172.21.15.46:41464) 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f484f640 1 -- 172.21.15.46:0/1301617564 learned_addr learned my addr 172.21.15.46:0/1301617564 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.236+0000 7fc1f5050640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc003ef0 0x7fc1f0146870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f5851640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f0147140 0x7fc1f01475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f484f640 1 -- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f0147140 msgr2=0x7fc1f01475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f0147140 0x7fc1f01475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f484f640 1 -- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc003ef0 msgr2=0x7fc1f0146870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc003ef0 0x7fc1f0146870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f484f640 1 -- 172.21.15.46:0/1301617564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f0147ec0 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f484f640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 0x7fc1f00af430 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc1e002c2d0 tx=0x7fc1e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0003810 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e0003970 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f01481a0 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0003ce0 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.237+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f0148760 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.238+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f0148c80 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.239+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc1e0040450 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.239+0000 7fc1e5ffb640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1d406ae40 0x7fc1d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.239+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc1e00ab040 con 0x7fc1f00aefd0 2023-12-16T21:14:31.241 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.240+0000 7fc1f5050640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1d406ae40 0x7fc1d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.242 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.240+0000 7fc1f5050640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1d406ae40 0x7fc1d406d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009ae0 tx=0x7fc1ec009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.243 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.244+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc1f0148c80 con 0x7fc1f00aefd0 2023-12-16T21:14:31.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.487+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fc1f0148c40 con 0x7fc1f00aefd0 2023-12-16T21:14:31.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.489+0000 7fc1e5ffb640 1 -- 172.21.15.46:0/1301617564 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc1e0076500 con 0x7fc1f00aefd0 2023-12-16T21:14:31.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1d406ae40 msgr2=0x7fc1d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1d406ae40 0x7fc1d406d300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009ae0 tx=0x7fc1ec009190 comp rx=0 tx=0).stop 2023-12-16T21:14:31.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 msgr2=0x7fc1f00af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 0x7fc1f00af430 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc1e002c2d0 tx=0x7fc1e0004730 comp rx=0 tx=0).stop 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 shutdown_connections 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1d406ae40 0x7fc1d406d300 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1f00aefd0 0x7fc1f00af430 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1f0147140 0x7fc1f01475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 --2- 172.21.15.46:0/1301617564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1dc003ef0 0x7fc1f0146870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 >> 172.21.15.46:0/1301617564 conn(0x7fc1f0070e70 msgr2=0x7fc1f0158580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 shutdown_connections 2023-12-16T21:14:31.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.492+0000 7fc1f72b4640 1 -- 172.21.15.46:0/1301617564 wait complete. 2023-12-16T21:14:31.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-12-16T21:14:31.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.611+0000 7f27a9790640 1 Processor -- start 2023-12-16T21:14:31.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.611+0000 7f27a9790640 1 -- start start 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a9790640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27a4146460 0x7f27a40b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a9790640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f27a406f1c0 0x7f27a406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a9790640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a4146fd0 con 0x7f27a4146840 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a9790640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a406c180 con 0x7f27a4146460 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a9790640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a406c460 con 0x7f27a406f1c0 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a2ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f27a4146460 0x7f27a40b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f279bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f27a4146840 0x7f27a406eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36510/0 (socket says 172.21.15.46:36510) 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f279bfff640 1 -- 172.21.15.46:0/689991442 learned_addr learned my addr 172.21.15.46:0/689991442 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a2ffd640 1 -- 172.21.15.46:0/689991442 >> v2:172.21.15.121:3300/0 conn(0x7f27a406f1c0 msgr2=0x7f27a406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a2ffd640 1 --2- 172.21.15.46:0/689991442 >> v2:172.21.15.121:3300/0 conn(0x7f27a406f1c0 0x7f27a406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a2ffd640 1 -- 172.21.15.46:0/689991442 >> v1:172.21.15.46:6789/0 conn(0x7f27a4146840 legacy=0x7f27a406eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a2ffd640 1 -- 172.21.15.46:0/689991442 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a406c740 con 0x7f27a4146460 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.612+0000 7f27a27fc640 1 -- 172.21.15.46:0/689991442 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675940858 0 0) 0x7f27a4146fd0 con 0x7f27a4146840 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.613+0000 7f27a2ffd640 1 --2- 172.21.15.46:0/689991442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146460 0x7f27a40b49d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f27a40b4fe0 tx=0x7f279402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2edc215430d8d62 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.613+0000 7f27a27fc640 1 -- 172.21.15.46:0/689991442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794031640 con 0x7f27a4146460 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.613+0000 7f27a27fc640 1 -- 172.21.15.46:0/689991442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27940317a0 con 0x7f27a4146460 2023-12-16T21:14:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.613+0000 7f27a27fc640 1 -- 172.21.15.46:0/689991442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794031b30 con 0x7f27a4146460 2023-12-16T21:14:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 -- 172.21.15.46:0/689991442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146460 msgr2=0x7f27a40b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 --2- 172.21.15.46:0/689991442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146460 0x7f27a40b49d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f27a40b4fe0 tx=0x7f279402b8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 -- 172.21.15.46:0/689991442 shutdown_connections 2023-12-16T21:14:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 --2- 172.21.15.46:0/689991442 >> v2:172.21.15.121:3300/0 conn(0x7f27a406f1c0 0x7f27a406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 --2- 172.21.15.46:0/689991442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146460 0x7f27a40b49d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 -- 172.21.15.46:0/689991442 >> 172.21.15.46:0/689991442 conn(0x7f27a406a6b0 msgr2=0x7f27a406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 -- 172.21.15.46:0/689991442 shutdown_connections 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.614+0000 7f27a9790640 1 -- 172.21.15.46:0/689991442 wait complete. 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.615+0000 7f27a9790640 1 Processor -- start 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.615+0000 7f27a9790640 1 -- start start 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a9790640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27a406f1c0 0x7f27a415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a9790640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27a4146460 0x7f27a40cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a9790640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 0x7f27a40a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a9790640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a413a8d0 con 0x7f27a4146840 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a9790640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a415d560 con 0x7f27a406f1c0 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a9790640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a41f7240 con 0x7f27a4146460 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 0x7f27a40a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 0x7f27a40a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41474/0 (socket says 172.21.15.46:41474) 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 -- 172.21.15.46:0/570696081 learned_addr learned my addr 172.21.15.46:0/570696081 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f279bfff640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27a4146460 0x7f27a40cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a2ffd640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27a406f1c0 0x7f27a415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 -- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27a4146460 msgr2=0x7f27a40cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27a4146460 0x7f27a40cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 -- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27a406f1c0 msgr2=0x7f27a415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27a406f1c0 0x7f27a415d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 -- 172.21.15.46:0/570696081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2794009400 con 0x7f27a4146840 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a2ffd640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27a406f1c0 0x7f27a415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f279bfff640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27a4146460 0x7f27a40cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.616+0000 7f27a37fe640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 0x7f27a40a9550 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f279c00ba10 tx=0x7f279c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.617+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c00cac0 con 0x7f27a4146840 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.617+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f279c00cc40 con 0x7f27a4146840 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.617+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a41f6f60 con 0x7f27a4146840 2023-12-16T21:14:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.617+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279c014440 con 0x7f27a4146840 2023-12-16T21:14:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.617+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a40cff40 con 0x7f27a4146840 2023-12-16T21:14:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.618+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a4069400 con 0x7f27a4146840 2023-12-16T21:14:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.618+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f279c01d3b0 con 0x7f27a4146840 2023-12-16T21:14:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.619+0000 7f279b7fe640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f278406ae40 0x7f278406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.619+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f279c089100 con 0x7f27a4146840 2023-12-16T21:14:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.619+0000 7f27a2ffd640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f278406ae40 0x7f278406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.620+0000 7f27a2ffd640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f278406ae40 0x7f278406d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2794036000 tx=0x7f279403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.623+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f279c054590 con 0x7f27a4146840 2023-12-16T21:14:31.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.867+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f27a4070460 con 0x7f27a4146840 2023-12-16T21:14:31.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.869+0000 7f279b7fe640 1 -- 172.21.15.46:0/570696081 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f279c057bb0 con 0x7f27a4146840 2023-12-16T21:14:31.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.870+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f278406ae40 msgr2=0x7f278406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.870+0000 7f27a9790640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f278406ae40 0x7f278406d300 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2794036000 tx=0x7f279403c040 comp rx=0 tx=0).stop 2023-12-16T21:14:31.870 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.870+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 msgr2=0x7f27a40a9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.870+0000 7f27a9790640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 0x7f27a40a9550 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f279c00ba10 tx=0x7f279c00bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 shutdown_connections 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f278406ae40 0x7f278406d300 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f27a4146840 0x7f27a40a9550 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27a4146460 0x7f27a40cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 --2- 172.21.15.46:0/570696081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f27a406f1c0 0x7f27a415d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 >> 172.21.15.46:0/570696081 conn(0x7f27a406a6b0 msgr2=0x7f27a4145a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 shutdown_connections 2023-12-16T21:14:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.871+0000 7f27a9790640 1 -- 172.21.15.46:0/570696081 wait complete. 2023-12-16T21:14:31.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-12-16T21:14:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.989+0000 7f5724623640 1 Processor -- start 2023-12-16T21:14:31.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.989+0000 7f5724623640 1 -- start start 2023-12-16T21:14:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5724623640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c15d2d0 0x7f571c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5724623640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f571c160f70 0x7f571c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5724623640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571c1614e0 con 0x7f571c0ce480 2023-12-16T21:14:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5724623640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571c1617c0 con 0x7f571c160f70 2023-12-16T21:14:31.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5724623640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c161aa0 con 0x7f571c15d2d0 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f571c160f70 0x7f571c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f571c160f70 0x7f571c0cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41482/0 (socket says 172.21.15.46:41482) 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 -- 172.21.15.46:0/1835013605 learned_addr learned my addr 172.21.15.46:0/1835013605 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f57223bf640 1 --2- 172.21.15.46:0/1835013605 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c15d2d0 0x7f571c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 -- 172.21.15.46:0/1835013605 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c15d2d0 msgr2=0x7f571c0b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 --2- 172.21.15.46:0/1835013605 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c15d2d0 0x7f571c0b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 -- 172.21.15.46:0/1835013605 >> v1:172.21.15.121:6789/0 conn(0x7f571c0ce480 legacy=0x7f571c06b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.990+0000 7f5721bbe640 1 -- 172.21.15.46:0/1835013605 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571c161d80 con 0x7f571c160f70 2023-12-16T21:14:31.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.991+0000 7f5721bbe640 1 --2- 172.21.15.46:0/1835013605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c160f70 0x7f571c0cdf40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5710009870 tx=0x7f571002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0bb00f3e78e3711 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.991+0000 7f57213bd640 1 -- 172.21.15.46:0/1835013605 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710030840 con 0x7f571c160f70 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.991+0000 7f57213bd640 1 -- 172.21.15.46:0/1835013605 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57100309a0 con 0x7f571c160f70 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.991+0000 7f57213bd640 1 -- 172.21.15.46:0/1835013605 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710030d30 con 0x7f571c160f70 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 -- 172.21.15.46:0/1835013605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c160f70 msgr2=0x7f571c0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 --2- 172.21.15.46:0/1835013605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c160f70 0x7f571c0cdf40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5710009870 tx=0x7f571002b980 comp rx=0 tx=0).stop 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 -- 172.21.15.46:0/1835013605 shutdown_connections 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 --2- 172.21.15.46:0/1835013605 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c160f70 0x7f571c0cdf40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 --2- 172.21.15.46:0/1835013605 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c15d2d0 0x7f571c0b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 -- 172.21.15.46:0/1835013605 >> 172.21.15.46:0/1835013605 conn(0x7f571c072230 msgr2=0x7f571c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 -- 172.21.15.46:0/1835013605 shutdown_connections 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.992+0000 7f5724623640 1 -- 172.21.15.46:0/1835013605 wait complete. 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.993+0000 7f5724623640 1 Processor -- start 2023-12-16T21:14:31.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.993+0000 7f5724623640 1 -- start start 2023-12-16T21:14:31.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5724623640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c0ce480 0x7f571c160130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5724623640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 0x7f571c0c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5724623640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f571c160f70 0x7f571c148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5724623640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c0c6ad0 con 0x7f571c15d2d0 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5724623640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571c156c30 con 0x7f571c160f70 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5724623640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571c15f090 con 0x7f571c0ce480 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 0x7f571c0c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 0x7f571c0c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41486/0 (socket says 172.21.15.46:41486) 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 -- 172.21.15.46:0/1513251214 learned_addr learned my addr 172.21.15.46:0/1513251214 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f57223bf640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c0ce480 0x7f571c160130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5722bc0640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f571c160f70 0x7f571c148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 -- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c0ce480 msgr2=0x7f571c160130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c0ce480 0x7f571c160130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 -- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f571c160f70 msgr2=0x7f571c148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f571c160f70 0x7f571c148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 -- 172.21.15.46:0/1513251214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57100094d0 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f57223bf640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c0ce480 0x7f571c160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5722bc0640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f571c160f70 0x7f571c148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.994+0000 7f5721bbe640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 0x7f571c0c6590 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f571002b9c0 tx=0x7f57100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.995+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571003e070 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.995+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710039a10 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.995+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571c15f370 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.995+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710039de0 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.995+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571c15f930 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.996+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571c069400 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.996+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f571004b400 con 0x7f571c15d2d0 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.997+0000 7f571affd640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f570806ae40 0x7f570806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:31.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.997+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f57100ab910 con 0x7f571c15d2d0 2023-12-16T21:14:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.997+0000 7f57223bf640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f570806ae40 0x7f570806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:31.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:31.997+0000 7f57223bf640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f570806ae40 0x7f570806d300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5714009b20 tx=0x7f57140039d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.001+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5710076d10 con 0x7f571c15d2d0 2023-12-16T21:14:32.246 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.246+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f571c0d0aa0 con 0x7f571c15d2d0 2023-12-16T21:14:32.247 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.248+0000 7f571affd640 1 -- 172.21.15.46:0/1513251214 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v11) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5710034070 con 0x7f571c15d2d0 2023-12-16T21:14:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.250+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f570806ae40 msgr2=0x7f570806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.250+0000 7f5724623640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f570806ae40 0x7f570806d300 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5714009b20 tx=0x7f57140039d0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.250+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 msgr2=0x7f571c0c6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.250+0000 7f5724623640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 0x7f571c0c6590 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f571002b9c0 tx=0x7f57100046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 shutdown_connections 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f570806ae40 0x7f570806d300 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f571c15d2d0 0x7f571c0c6590 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f571c160f70 0x7f571c148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 --2- 172.21.15.46:0/1513251214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f571c0ce480 0x7f571c160130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 >> 172.21.15.46:0/1513251214 conn(0x7f571c072230 msgr2=0x7f571c0c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 shutdown_connections 2023-12-16T21:14:32.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.251+0000 7f5724623640 1 -- 172.21.15.46:0/1513251214 wait complete. 2023-12-16T21:14:32.261 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-12-16T21:14:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.369+0000 7fe27741c640 1 Processor -- start 2023-12-16T21:14:32.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.370+0000 7fe27741c640 1 -- start start 2023-12-16T21:14:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.370+0000 7fe27741c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2700d2910 0x7fe2700d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.370+0000 7fe27741c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700b2b10 con 0x7fe2700d3730 2023-12-16T21:14:32.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.370+0000 7fe27741c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700d1590 con 0x7fe27015d330 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.370+0000 7fe27741c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2700d3bc0 con 0x7fe2700d2910 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.370+0000 7fe2749b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2700d2910 0x7fe2700d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe2759b9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe2700d3730 0x7fe2700b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36512/0 (socket says 172.21.15.46:36512) 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe2749b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe2700d2910 0x7fe2700d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41498/0 (socket says 172.21.15.46:41498) 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe2759b9640 1 -- 172.21.15.46:0/4159482070 learned_addr learned my addr 172.21.15.46:0/4159482070 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe2749b7640 1 -- 172.21.15.46:0/4159482070 >> v1:172.21.15.121:6789/0 conn(0x7fe27015d330 legacy=0x7fe2700b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe2749b7640 1 -- 172.21.15.46:0/4159482070 >> v1:172.21.15.46:6789/0 conn(0x7fe2700d3730 legacy=0x7fe2700b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe26ffff640 1 -- 172.21.15.46:0/4159482070 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538831995 0 0) 0x7fe2700b2b10 con 0x7fe2700d3730 2023-12-16T21:14:32.371 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.371+0000 7fe2749b7640 1 -- 172.21.15.46:0/4159482070 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2700d1870 con 0x7fe2700d2910 2023-12-16T21:14:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe2749b7640 1 --2- 172.21.15.46:0/4159482070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2700d2910 0x7fe2700d31f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe2640097a0 tx=0x7fe26402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e70bca3451e4ce29 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe26ffff640 1 -- 172.21.15.46:0/4159482070 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264031830 con 0x7fe2700d2910 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe26ffff640 1 -- 172.21.15.46:0/4159482070 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe264031990 con 0x7fe2700d2910 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe26ffff640 1 -- 172.21.15.46:0/4159482070 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264031d20 con 0x7fe2700d2910 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 -- 172.21.15.46:0/4159482070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2700d2910 msgr2=0x7fe2700d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 --2- 172.21.15.46:0/4159482070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2700d2910 0x7fe2700d31f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe2640097a0 tx=0x7fe26402b8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 -- 172.21.15.46:0/4159482070 shutdown_connections 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 --2- 172.21.15.46:0/4159482070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe2700d2910 0x7fe2700d31f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 -- 172.21.15.46:0/4159482070 >> 172.21.15.46:0/4159482070 conn(0x7fe270072020 msgr2=0x7fe270072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 -- 172.21.15.46:0/4159482070 shutdown_connections 2023-12-16T21:14:32.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.372+0000 7fe27741c640 1 -- 172.21.15.46:0/4159482070 wait complete. 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 Processor -- start 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 -- start start 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2700d2910 0x7fe270150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2700d3730 0x7fe27015f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 0x7fe2700cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27015f810 con 0x7fe27015d330 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27015faf0 con 0x7fe2700d3730 2023-12-16T21:14:32.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe27741c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe27015fdd0 con 0x7fe2700d2910 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe2759b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 0x7fe2700cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe2759b9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 0x7fe2700cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41504/0 (socket says 172.21.15.46:41504) 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.374+0000 7fe2759b9640 1 -- 172.21.15.46:0/3531305934 learned_addr learned my addr 172.21.15.46:0/3531305934 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2749b7640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2700d3730 0x7fe27015f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2751b8640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2700d2910 0x7fe270150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2759b9640 1 -- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2700d2910 msgr2=0x7fe270150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2759b9640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2700d2910 0x7fe270150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2759b9640 1 -- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2700d3730 msgr2=0x7fe27015f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2759b9640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2700d3730 0x7fe27015f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2759b9640 1 -- 172.21.15.46:0/3531305934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe264009400 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe2759b9640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 0x7fe2700cd260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe26800c450 tx=0x7fe26800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2680074e0 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268007660 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2701600b0 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe268004a20 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.375+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2701608b0 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.376+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2701605d0 con 0x7fe27015d330 2023-12-16T21:14:32.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.377+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe268004b80 con 0x7fe27015d330 2023-12-16T21:14:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.377+0000 7fe26dffb640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe25c06ae40 0x7fe25c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.377+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe2680883c0 con 0x7fe27015d330 2023-12-16T21:14:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.377+0000 7fe2751b8640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe25c06ae40 0x7fe25c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.378+0000 7fe2751b8640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe25c06ae40 0x7fe25c06d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe2600012e0 tx=0x7fe260009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.381+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe2701605d0 con 0x7fe27015d330 2023-12-16T21:14:32.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.624+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe2701605d0 con 0x7fe27015d330 2023-12-16T21:14:32.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.626+0000 7fe26dffb640 1 -- 172.21.15.46:0/3531305934 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe2701605d0 con 0x7fe27015d330 2023-12-16T21:14:32.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe25c06ae40 msgr2=0x7fe25c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe25c06ae40 0x7fe25c06d300 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe2600012e0 tx=0x7fe260009040 comp rx=0 tx=0).stop 2023-12-16T21:14:32.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 msgr2=0x7fe2700cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 0x7fe2700cd260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe26800c450 tx=0x7fe26800c910 comp rx=0 tx=0).stop 2023-12-16T21:14:32.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 shutdown_connections 2023-12-16T21:14:32.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe25c06ae40 0x7fe25c06d300 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe27015d330 0x7fe2700cd260 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2700d3730 0x7fe27015f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 --2- 172.21.15.46:0/3531305934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2700d2910 0x7fe270150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 >> 172.21.15.46:0/3531305934 conn(0x7fe270072020 msgr2=0x7fe270072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:32.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 shutdown_connections 2023-12-16T21:14:32.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.628+0000 7fe27741c640 1 -- 172.21.15.46:0/3531305934 wait complete. 2023-12-16T21:14:32.639 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-12-16T21:14:32.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.748+0000 7f8bb2dbe640 1 Processor -- start 2023-12-16T21:14:32.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.748+0000 7f8bb2dbe640 1 -- start start 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb2dbe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8bac15d2d0 0x7f8bac0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb2dbe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8bac160f70 0x7f8bac0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb2dbe640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac1614e0 con 0x7f8bac0ce480 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb2dbe640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac1617c0 con 0x7f8bac160f70 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb2dbe640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac161aa0 con 0x7f8bac15d2d0 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8bac160f70 0x7f8bac0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8bac160f70 0x7f8bac0cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41520/0 (socket says 172.21.15.46:41520) 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 -- 172.21.15.46:0/2312832877 learned_addr learned my addr 172.21.15.46:0/2312832877 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:32.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb135b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8bac0ce480 0x7f8bac06b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36520/0 (socket says 172.21.15.46:36520) 2023-12-16T21:14:32.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8bb0b5a640 1 --2- 172.21.15.46:0/2312832877 >> v2:172.21.15.121:3300/0 conn(0x7f8bac15d2d0 0x7f8bac0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 -- 172.21.15.46:0/2312832877 >> v2:172.21.15.121:3300/0 conn(0x7f8bac15d2d0 msgr2=0x7f8bac0b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:32.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 --2- 172.21.15.46:0/2312832877 >> v2:172.21.15.121:3300/0 conn(0x7f8bac15d2d0 0x7f8bac0b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 -- 172.21.15.46:0/2312832877 >> v1:172.21.15.46:6789/0 conn(0x7f8bac0ce480 legacy=0x7f8bac06b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.749+0000 7f8babfff640 1 -- 172.21.15.46:0/2312832877 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac161d80 con 0x7f8bac160f70 2023-12-16T21:14:32.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.750+0000 7f8babfff640 1 --2- 172.21.15.46:0/2312832877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac160f70 0x7f8bac0cdf40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0b5db0 tx=0x7f8b9802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9047ff168efa7f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.750+0000 7f8bab7fe640 1 -- 172.21.15.46:0/2312832877 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98030860 con 0x7f8bac160f70 2023-12-16T21:14:32.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.750+0000 7f8bab7fe640 1 -- 172.21.15.46:0/2312832877 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b980309c0 con 0x7f8bac160f70 2023-12-16T21:14:32.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.750+0000 7f8bab7fe640 1 -- 172.21.15.46:0/2312832877 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98030d50 con 0x7f8bac160f70 2023-12-16T21:14:32.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/2312832877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac160f70 msgr2=0x7f8bac0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/2312832877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac160f70 0x7f8bac0cdf40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0b5db0 tx=0x7f8b9802b9a0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/2312832877 shutdown_connections 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/2312832877 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac160f70 0x7f8bac0cdf40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/2312832877 >> v2:172.21.15.121:3300/0 conn(0x7f8bac15d2d0 0x7f8bac0b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/2312832877 >> 172.21.15.46:0/2312832877 conn(0x7f8bac072230 msgr2=0x7f8bac0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/2312832877 shutdown_connections 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.751+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/2312832877 wait complete. 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.752+0000 7f8bb2dbe640 1 Processor -- start 2023-12-16T21:14:32.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.752+0000 7f8bb2dbe640 1 -- start start 2023-12-16T21:14:32.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bac0ce480 0x7f8bac152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 0x7f8bac15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bac160f70 0x7f8bac15f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac157340 con 0x7f8bac15d2d0 2023-12-16T21:14:32.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac15f340 con 0x7f8bac160f70 2023-12-16T21:14:32.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac1575e0 con 0x7f8bac0ce480 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 0x7f8bac15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 0x7f8bac15e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41530/0 (socket says 172.21.15.46:41530) 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 -- 172.21.15.46:0/3645654174 learned_addr learned my addr 172.21.15.46:0/3645654174 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb0b5a640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bac0ce480 0x7f8bac152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 -- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bac0ce480 msgr2=0x7f8bac152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bac0ce480 0x7f8bac152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 -- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bac160f70 msgr2=0x7f8bac15f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bac160f70 0x7f8bac15f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 -- 172.21.15.46:0/3645654174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b980094d0 con 0x7f8bac15d2d0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8babfff640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 0x7f8bac15e960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b9802b9e0 tx=0x7f8b980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9803e070 con 0x7f8bac15d2d0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b98039a60 con 0x7f8bac15d2d0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac1578c0 con 0x7f8bac15d2d0 2023-12-16T21:14:32.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b98039df0 con 0x7f8bac15d2d0 2023-12-16T21:14:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.753+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac152980 con 0x7f8bac15d2d0 2023-12-16T21:14:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.754+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac157de0 con 0x7f8bac15d2d0 2023-12-16T21:14:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.755+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8b9804b400 con 0x7f8bac15d2d0 2023-12-16T21:14:32.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.755+0000 7f8ba97fa640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b9006ae40 0x7f8b9006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.755+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8b980aba20 con 0x7f8bac15d2d0 2023-12-16T21:14:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.756+0000 7f8bb0b5a640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b9006ae40 0x7f8b9006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:32.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.756+0000 7f8bb0b5a640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b9006ae40 0x7f8b9006d300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0006fd0 tx=0x7f8ba0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:32.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:32.760+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8bac157de0 con 0x7f8bac15d2d0 2023-12-16T21:14:33.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.003+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f8bac157de0 con 0x7f8bac15d2d0 2023-12-16T21:14:33.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.005+0000 7f8ba97fa640 1 -- 172.21.15.46:0/3645654174 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v13) v1 ==== 97+0+0 (secure 0 0 0) 0x7f8bac157de0 con 0x7f8bac15d2d0 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b9006ae40 msgr2=0x7f8b9006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b9006ae40 0x7f8b9006d300 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0006fd0 tx=0x7f8ba0008040 comp rx=0 tx=0).stop 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 msgr2=0x7f8bac15e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 0x7f8bac15e960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b9802b9e0 tx=0x7f8b980046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 shutdown_connections 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b9006ae40 0x7f8b9006d300 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bac15d2d0 0x7f8bac15e960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bac160f70 0x7f8bac15f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 --2- 172.21.15.46:0/3645654174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bac0ce480 0x7f8bac152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 >> 172.21.15.46:0/3645654174 conn(0x7f8bac072230 msgr2=0x7f8bac0b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:33.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 shutdown_connections 2023-12-16T21:14:33.011 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.007+0000 7f8bb2dbe640 1 -- 172.21.15.46:0/3645654174 wait complete. 2023-12-16T21:14:33.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-12-16T21:14:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.126+0000 7fea21d1d640 1 Processor -- start 2023-12-16T21:14:33.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.126+0000 7fea21d1d640 1 -- start start 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.127+0000 7fea21d1d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 0x7fea1c167df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.127+0000 7fea21d1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c0d1bb0 con 0x7fea1c15d2d0 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.127+0000 7fea21d1d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c15d6b0 con 0x7fea1c0b5370 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.127+0000 7fea21d1d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c0ce700 con 0x7fea1c0d2040 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.127+0000 7fea1b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fea1c15d2d0 0x7fea1c0b4c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36534/0 (socket says 172.21.15.46:36534) 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.127+0000 7fea1b7fe640 1 -- 172.21.15.46:0/1576260954 learned_addr learned my addr 172.21.15.46:0/1576260954 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 0x7fea1c167df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633941473 0 0) 0x7fea1c0d1bb0 con 0x7fea1c15d2d0 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea04003520 con 0x7fea1c15d2d0 2023-12-16T21:14:33.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3308107555 0 0) 0x7fea04003520 con 0x7fea1c15d2d0 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 msgr2=0x7fea1c167df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 0x7fea1c167df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 >> v1:172.21.15.121:6789/0 conn(0x7fea1c0b5370 legacy=0x7fea1c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c0d1280 con 0x7fea1c15d2d0 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fea0c001670 con 0x7fea1c15d2d0 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 >> v1:172.21.15.46:6789/0 conn(0x7fea1c15d2d0 legacy=0x7fea1c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 0x7fea040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea04003ee0 0x7fea040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea04004d00 0x7fea04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c0ce700 con 0x7fea04004d00 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0400b9e0 con 0x7fea04003ee0 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0400bba0 con 0x7fea1c0d2040 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea04004d00 0x7fea04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1b7fe640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 0x7fea040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1affd640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea04003ee0 0x7fea040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 -- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 msgr2=0x7fea040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c0d2040 0x7fea040039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 -- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea04003ee0 msgr2=0x7fea040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea04003ee0 0x7fea040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.128+0000 7fea1bfff640 1 -- 172.21.15.46:0/1576260954 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04003520 con 0x7fea04004d00 2023-12-16T21:14:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.129+0000 7fea1bfff640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea04004d00 0x7fea04005180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea14009fc0 tx=0x7fea040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e19c6bce438bf4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.129+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1402cc20 con 0x7fea04004d00 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.129+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1400f040 con 0x7fea04004d00 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.129+0000 7fea1a7fc640 1 -- 172.21.15.46:0/1576260954 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea14034440 con 0x7fea04004d00 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea21d1d640 1 -- 172.21.15.46:0/1576260954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea04004d00 msgr2=0x7fea04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea21d1d640 1 --2- 172.21.15.46:0/1576260954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea04004d00 0x7fea04005180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fea14009fc0 tx=0x7fea040057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea1bfff640 1 -- 172.21.15.46:0/1576260954 reap_dead start 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea21d1d640 1 -- 172.21.15.46:0/1576260954 shutdown_connections 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea21d1d640 1 -- 172.21.15.46:0/1576260954 >> 172.21.15.46:0/1576260954 conn(0x7fea1c072050 msgr2=0x7fea1c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea21d1d640 1 -- 172.21.15.46:0/1576260954 shutdown_connections 2023-12-16T21:14:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.130+0000 7fea21d1d640 1 -- 172.21.15.46:0/1576260954 wait complete. 2023-12-16T21:14:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.131+0000 7fea21d1d640 1 Processor -- start 2023-12-16T21:14:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.131+0000 7fea21d1d640 1 -- start start 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea21d1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 0x7fea1c1f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea21d1d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea1c15b3a0 0x7fea1c15b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea21d1d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c1e7880 0x7fea1c1e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea21d1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1e88b0 con 0x7fea1c1f5960 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea21d1d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c15bd60 con 0x7fea1c15b3a0 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea21d1d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1f3090 con 0x7fea1c1e7880 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 0x7fea1c1f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 0x7fea1c1f2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41542/0 (socket says 172.21.15.46:41542) 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 -- 172.21.15.46:0/2239419004 learned_addr learned my addr 172.21.15.46:0/2239419004 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 -- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c1e7880 msgr2=0x7fea1c1e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c1e7880 0x7fea1c1e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 -- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea1c15b3a0 msgr2=0x7fea1c15b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea1c15b3a0 0x7fea1c15b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.132+0000 7fea1b7fe640 1 -- 172.21.15.46:0/2239419004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c1e85d0 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.133+0000 7fea1b7fe640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 0x7fea1c1f2b50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea0c007290 tx=0x7fea0c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.133+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c00f860 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.133+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c00f9c0 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.133+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c15d850 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.133+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c00fd70 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.133+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c15de10 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.134+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c15d6b0 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.134+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fea0c01d460 con 0x7fea1c1f5960 2023-12-16T21:14:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.135+0000 7fea03fff640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fea0806ae40 0x7fea0806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.135+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fea0c089420 con 0x7fea1c1f5960 2023-12-16T21:14:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.135+0000 7fea1affd640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fea0806ae40 0x7fea0806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.136+0000 7fea1affd640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fea0806ae40 0x7fea0806d300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0b3960 tx=0x7fea10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.139+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fea1c15d6b0 con 0x7fea1c1f5960 2023-12-16T21:14:33.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.381+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fea1c15d6b0 con 0x7fea1c1f5960 2023-12-16T21:14:33.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.383+0000 7fea03fff640 1 -- 172.21.15.46:0/2239419004 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v14) v1 ==== 90+0+0 (secure 0 0 0) 0x7fea1c15d6b0 con 0x7fea1c1f5960 2023-12-16T21:14:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fea0806ae40 msgr2=0x7fea0806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fea0806ae40 0x7fea0806d300 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fea1c0b3960 tx=0x7fea10009040 comp rx=0 tx=0).stop 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 msgr2=0x7fea1c1f2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 0x7fea1c1f2b50 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea0c007290 tx=0x7fea0c007750 comp rx=0 tx=0).stop 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 shutdown_connections 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fea0806ae40 0x7fea0806d300 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea1c1e7880 0x7fea1c1e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea1c15b3a0 0x7fea1c15b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 --2- 172.21.15.46:0/2239419004 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea1c1f5960 0x7fea1c1f2b50 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 >> 172.21.15.46:0/2239419004 conn(0x7fea1c072050 msgr2=0x7fea1c0d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 shutdown_connections 2023-12-16T21:14:33.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.385+0000 7fea21d1d640 1 -- 172.21.15.46:0/2239419004 wait complete. 2023-12-16T21:14:33.396 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:33.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.503+0000 7f105d5b0640 1 Processor -- start 2023-12-16T21:14:33.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f105d5b0640 1 -- start start 2023-12-16T21:14:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f105d5b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 0x7f10580d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f105d5b0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580b2b10 con 0x7f10580d3730 2023-12-16T21:14:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f105d5b0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580d3b50 con 0x7f105815d330 2023-12-16T21:14:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f105d5b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10580d1590 con 0x7f10580d2910 2023-12-16T21:14:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f10577fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f10580d3730 0x7f10580b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36536/0 (socket says 172.21.15.46:36536) 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.504+0000 7f10577fe640 1 -- 172.21.15.46:0/3275457911 learned_addr learned my addr 172.21.15.46:0/3275457911 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f10567fc640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 0x7f10580d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427785018 0 0) 0x7f10580b2b10 con 0x7f10580d3730 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1044003520 con 0x7f10580d3730 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512231610 0 0) 0x7f10580d3b50 con 0x7f105815d330 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10580b2b10 con 0x7f105815d330 2023-12-16T21:14:33.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924668877 0 0) 0x7f1044003520 con 0x7f10580d3730 2023-12-16T21:14:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 msgr2=0x7f10580d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 0x7f10580d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 >> v1:172.21.15.121:6789/0 conn(0x7f105815d330 legacy=0x7f10580b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10580d1870 con 0x7f10580d3730 2023-12-16T21:14:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f10480019f0 con 0x7f10580d3730 2023-12-16T21:14:33.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 >> v1:172.21.15.46:6789/0 conn(0x7f10580d3730 legacy=0x7f10580b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 0x7f1044003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440040e0 0x7f10440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1044004d00 0x7f1044005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10580d1590 con 0x7f1044004d00 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104400bb40 con 0x7f10440040e0 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104400bd00 con 0x7f10580d2910 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f10577fe640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1044004d00 0x7f1044005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f1056ffd640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 0x7f1044003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.505+0000 7f10567fc640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440040e0 0x7f10440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f10577fe640 1 -- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 msgr2=0x7f1044003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f10577fe640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10580d2910 0x7f1044003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f10577fe640 1 -- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440040e0 msgr2=0x7f10440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f10577fe640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10440040e0 0x7f10440047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f10577fe640 1 -- 172.21.15.46:0/3275457911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044003520 con 0x7f1044004d00 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f10577fe640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1044004d00 0x7f1044005180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f104800a710 tx=0x7f10440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1b0d03377d5ba7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048034630 con 0x7f1044004d00 2023-12-16T21:14:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104802f070 con 0x7f1044004d00 2023-12-16T21:14:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.506+0000 7f1055ffb640 1 -- 172.21.15.46:0/3275457911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048034a50 con 0x7f1044004d00 2023-12-16T21:14:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f105d5b0640 1 -- 172.21.15.46:0/3275457911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1044004d00 msgr2=0x7f1044005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f105d5b0640 1 --2- 172.21.15.46:0/3275457911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1044004d00 0x7f1044005180 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f104800a710 tx=0x7f10440057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f10577fe640 1 -- 172.21.15.46:0/3275457911 reap_dead start 2023-12-16T21:14:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f105d5b0640 1 -- 172.21.15.46:0/3275457911 shutdown_connections 2023-12-16T21:14:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f105d5b0640 1 -- 172.21.15.46:0/3275457911 >> 172.21.15.46:0/3275457911 conn(0x7f1058072020 msgr2=0x7f1058072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f105d5b0640 1 -- 172.21.15.46:0/3275457911 shutdown_connections 2023-12-16T21:14:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.507+0000 7f105d5b0640 1 -- 172.21.15.46:0/3275457911 wait complete. 2023-12-16T21:14:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.508+0000 7f105d5b0640 1 Processor -- start 2023-12-16T21:14:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.508+0000 7f105d5b0640 1 -- start start 2023-12-16T21:14:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f105d5b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1058145a00 0x7f105815e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f105d5b0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1058152690 0x7f1058145de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f105d5b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 0x7f10581f30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f105d5b0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10581558e0 con 0x7f1058146320 2023-12-16T21:14:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f105d5b0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058155310 con 0x7f1058152690 2023-12-16T21:14:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f105d5b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1058153610 con 0x7f1058145a00 2023-12-16T21:14:33.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f10577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 0x7f10581f30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f10577fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 0x7f10581f30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41560/0 (socket says 172.21.15.46:41560) 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f10577fe640 1 -- 172.21.15.46:0/4006703963 learned_addr learned my addr 172.21.15.46:0/4006703963 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f10567fc640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1058152690 0x7f1058145de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.509+0000 7f1056ffd640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1058145a00 0x7f105815e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10577fe640 1 -- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1058145a00 msgr2=0x7f105815e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10577fe640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1058145a00 0x7f105815e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10577fe640 1 -- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1058152690 msgr2=0x7f1058145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10577fe640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1058152690 0x7f1058145de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10577fe640 1 -- 172.21.15.46:0/4006703963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105815eca0 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10567fc640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1058152690 0x7f1058145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f10577fe640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 0x7f10581f30b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1048000f80 tx=0x7f1048001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048034630 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1048034790 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048034b00 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1058153860 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.510+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1058155500 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.511+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058155a20 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.511+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1048031030 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.512+0000 7f104f7fe640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f103806ae40 0x7f103806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.512+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f10480ab470 con 0x7f1058146320 2023-12-16T21:14:33.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.512+0000 7f1056ffd640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f103806ae40 0x7f103806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.513+0000 7f1056ffd640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f103806ae40 0x7f103806d300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10580c6ef0 tx=0x7f1050009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.516+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1058155a20 con 0x7f1058146320 2023-12-16T21:14:33.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.693+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1058155a20 con 0x7f1058146320 2023-12-16T21:14:33.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.694+0000 7f104f7fe640 1 -- 172.21.15.46:0/4006703963 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12857 (secure 0 0 0) 0x7f1058155a20 con 0x7f1058146320 2023-12-16T21:14:33.693 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:33.693 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":30,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:30.731002+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:33.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f103806ae40 msgr2=0x7f103806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f103806ae40 0x7f103806d300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10580c6ef0 tx=0x7f1050009040 comp rx=0 tx=0).stop 2023-12-16T21:14:33.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 msgr2=0x7f10581f30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 0x7f10581f30b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1048000f80 tx=0x7f1048001cb0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 shutdown_connections 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f103806ae40 0x7f103806d300 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1058146320 0x7f10581f30b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1058152690 0x7f1058145de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 --2- 172.21.15.46:0/4006703963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1058145a00 0x7f105815e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.695+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 >> 172.21.15.46:0/4006703963 conn(0x7f1058072020 msgr2=0x7f105815a110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.696+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 shutdown_connections 2023-12-16T21:14:33.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.696+0000 7f105d5b0640 1 -- 172.21.15.46:0/4006703963 wait complete. 2023-12-16T21:14:33.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:33.707 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:33.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.817+0000 7f7663270640 1 Processor -- start 2023-12-16T21:14:33.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.817+0000 7f7663270640 1 -- start start 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f7663270640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f765c143490 0x7f765c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f7663270640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f765c15abf0 0x7f765c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f7663270640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f765c15e2a0 con 0x7f765c15e730 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f7663270640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c15b520 con 0x7f765c15abf0 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f7663270640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f765c143870 con 0x7f765c143490 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f765c15abf0 0x7f765c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f765c15abf0 0x7f765c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41564/0 (socket says 172.21.15.46:41564) 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 -- 172.21.15.46:0/1758145161 learned_addr learned my addr 172.21.15.46:0/1758145161 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:33.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766100c640 1 --2- 172.21.15.46:0/1758145161 >> v2:172.21.15.121:3300/0 conn(0x7f765c143490 0x7f765c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 -- 172.21.15.46:0/1758145161 >> v2:172.21.15.121:3300/0 conn(0x7f765c143490 msgr2=0x7f765c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 --2- 172.21.15.46:0/1758145161 >> v2:172.21.15.121:3300/0 conn(0x7f765c143490 0x7f765c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 -- 172.21.15.46:0/1758145161 >> v1:172.21.15.121:6789/0 conn(0x7f765c15e730 legacy=0x7f765c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766180d640 1 -- 172.21.15.46:0/1758145161 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c15b240 con 0x7f765c15abf0 2023-12-16T21:14:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.818+0000 7f766100c640 1 --2- 172.21.15.46:0/1758145161 >> v2:172.21.15.121:3300/0 conn(0x7f765c143490 0x7f765c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.819+0000 7f766180d640 1 --2- 172.21.15.46:0/1758145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c15abf0 0x7f765c0b1230 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f765c06e620 tx=0x7f765402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381d6acac6277c8a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.819+0000 7f7653fff640 1 -- 172.21.15.46:0/1758145161 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765402ca80 con 0x7f765c15abf0 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.819+0000 7f7653fff640 1 -- 172.21.15.46:0/1758145161 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765402cbe0 con 0x7f765c15abf0 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.819+0000 7f7653fff640 1 -- 172.21.15.46:0/1758145161 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540343f0 con 0x7f765c15abf0 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 -- 172.21.15.46:0/1758145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c15abf0 msgr2=0x7f765c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 --2- 172.21.15.46:0/1758145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c15abf0 0x7f765c0b1230 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f765c06e620 tx=0x7f765402bee0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 -- 172.21.15.46:0/1758145161 shutdown_connections 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 --2- 172.21.15.46:0/1758145161 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c15abf0 0x7f765c0b1230 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 --2- 172.21.15.46:0/1758145161 >> v2:172.21.15.121:3300/0 conn(0x7f765c143490 0x7f765c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 -- 172.21.15.46:0/1758145161 >> 172.21.15.46:0/1758145161 conn(0x7f765c06a4d0 msgr2=0x7f765c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 -- 172.21.15.46:0/1758145161 shutdown_connections 2023-12-16T21:14:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.820+0000 7f7663270640 1 -- 172.21.15.46:0/1758145161 wait complete. 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.821+0000 7f7663270640 1 Processor -- start 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.821+0000 7f7663270640 1 -- start start 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f7663270640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 0x7f765c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f7663270640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f765c15abf0 0x7f765c0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f7663270640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f765c15e730 0x7f765c15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f7663270640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1f4d60 con 0x7f765c143490 2023-12-16T21:14:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f7663270640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c159f30 con 0x7f765c15e730 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f7663270640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1f4190 con 0x7f765c15abf0 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 0x7f765c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 0x7f765c1599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41568/0 (socket says 172.21.15.46:41568) 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 -- 172.21.15.46:0/568157722 learned_addr learned my addr 172.21.15.46:0/568157722 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766080b640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f765c15abf0 0x7f765c0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 -- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f765c15abf0 msgr2=0x7f765c0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f765c15abf0 0x7f765c0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766180d640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f765c15e730 0x7f765c15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 -- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f765c15e730 msgr2=0x7f765c15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f765c15e730 0x7f765c15b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766100c640 1 -- 172.21.15.46:0/568157722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7654009b30 con 0x7f765c143490 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766080b640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f765c15abf0 0x7f765c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.822+0000 7f766180d640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f765c15e730 0x7f765c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.823+0000 7f766100c640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 0x7f765c1599f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f765800c820 tx=0x7f765800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.823+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658007a30 con 0x7f765c143490 2023-12-16T21:14:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.823+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658007b90 con 0x7f765c143490 2023-12-16T21:14:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.823+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765800ce70 con 0x7f765c143490 2023-12-16T21:14:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.823+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c1f4430 con 0x7f765c143490 2023-12-16T21:14:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.823+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c1f49f0 con 0x7f765c143490 2023-12-16T21:14:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.824+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c069400 con 0x7f765c143490 2023-12-16T21:14:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.825+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f76580113f0 con 0x7f765c143490 2023-12-16T21:14:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.825+0000 7f7651ffb640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f764006ae70 0x7f764006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.825+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7658089390 con 0x7f765c143490 2023-12-16T21:14:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.825+0000 7f766080b640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f764006ae70 0x7f764006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.826+0000 7f766080b640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f764006ae70 0x7f764006d330 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f765c0cb010 tx=0x7f764c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:33.829+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7658054820 con 0x7f765c143490 2023-12-16T21:14:34.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.005+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f765c0b2820 con 0x7f765c143490 2023-12-16T21:14:34.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.006+0000 7f7651ffb640 1 -- 172.21.15.46:0/568157722 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7658057e40 con 0x7f765c143490 2023-12-16T21:14:34.006 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f764006ae70 msgr2=0x7f764006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f764006ae70 0x7f764006d330 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f765c0cb010 tx=0x7f764c008040 comp rx=0 tx=0).stop 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 msgr2=0x7f765c1599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 0x7f765c1599f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f765800c820 tx=0x7f765800cc20 comp rx=0 tx=0).stop 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 shutdown_connections 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f764006ae70 0x7f764006d330 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f765c15e730 0x7f765c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f765c15abf0 0x7f765c0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 --2- 172.21.15.46:0/568157722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c143490 0x7f765c1599f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 >> 172.21.15.46:0/568157722 conn(0x7f765c06a4d0 msgr2=0x7f765c141b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 shutdown_connections 2023-12-16T21:14:34.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.008+0000 7f7663270640 1 -- 172.21.15.46:0/568157722 wait complete. 2023-12-16T21:14:34.019 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-16T21:14:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.126+0000 7f36ca675640 1 Processor -- start 2023-12-16T21:14:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.126+0000 7f36ca675640 1 -- start start 2023-12-16T21:14:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36ca675640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c4161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36ca675640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c4161d20 con 0x7f36c40d0ed0 2023-12-16T21:14:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36ca675640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c415e760 con 0x7f36c406ea50 2023-12-16T21:14:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36ca675640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c415d9a0 con 0x7f36c4160fb0 2023-12-16T21:14:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c8c12640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36c40d0ed0 0x7f36c40d1360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36550/0 (socket says 172.21.15.46:36550) 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c8c12640 1 -- 172.21.15.46:0/2945482539 learned_addr learned my addr 172.21.15.46:0/2945482539 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c37fe640 1 --2- 172.21.15.46:0/2945482539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c4161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873297222 0 0) 0x7f36c4161d20 con 0x7f36c40d0ed0 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b0003520 con 0x7f36c40d0ed0 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441678775 0 0) 0x7f36c415e760 con 0x7f36c406ea50 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.127+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c4161d20 con 0x7f36c406ea50 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.128+0000 7f36c37fe640 1 -- 172.21.15.46:0/2945482539 >> v1:172.21.15.121:6789/0 conn(0x7f36c406ea50 legacy=0x7f36c406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.128+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534780644 0 0) 0x7f36b0003520 con 0x7f36c40d0ed0 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.128+0000 7f36c37fe640 1 -- 172.21.15.46:0/2945482539 >> v1:172.21.15.46:6789/0 conn(0x7f36c40d0ed0 legacy=0x7f36c40d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.128+0000 7f36c37fe640 1 -- 172.21.15.46:0/2945482539 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c415dc80 con 0x7f36c4160fb0 2023-12-16T21:14:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.128+0000 7f36c37fe640 1 --2- 172.21.15.46:0/2945482539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c4161440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f36b40097d0 tx=0x7f36b402b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b287738deb4bd55d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc0019f0 con 0x7f36c4160fb0 2023-12-16T21:14:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b402e050 con 0x7f36c4160fb0 2023-12-16T21:14:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36c2ffd640 1 -- 172.21.15.46:0/2945482539 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4030730 con 0x7f36c4160fb0 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 -- 172.21.15.46:0/2945482539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 msgr2=0x7f36c4161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 --2- 172.21.15.46:0/2945482539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c4161440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f36b40097d0 tx=0x7f36b402b800 comp rx=0 tx=0).stop 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 -- 172.21.15.46:0/2945482539 shutdown_connections 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 --2- 172.21.15.46:0/2945482539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c4161440 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 -- 172.21.15.46:0/2945482539 >> 172.21.15.46:0/2945482539 conn(0x7f36c406a6b0 msgr2=0x7f36c406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 -- 172.21.15.46:0/2945482539 shutdown_connections 2023-12-16T21:14:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.129+0000 7f36ca675640 1 -- 172.21.15.46:0/2945482539 wait complete. 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.130+0000 7f36ca675640 1 Processor -- start 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 -- start start 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 0x7f36c4156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36c40d0ed0 0x7f36c4155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c413df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4156360 con 0x7f36c406ea50 2023-12-16T21:14:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c413ced0 con 0x7f36c40d0ed0 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36ca675640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c413d1b0 con 0x7f36c4160fb0 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 0x7f36c4156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36c3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 0x7f36c4156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41576/0 (socket says 172.21.15.46:41576) 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.131+0000 7f36c3fff640 1 -- 172.21.15.46:0/1313620627 learned_addr learned my addr 172.21.15.46:0/1313620627 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c8c12640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c413df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c37fe640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36c40d0ed0 0x7f36c4155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c3fff640 1 -- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 msgr2=0x7f36c413df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c3fff640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c413df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c3fff640 1 -- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36c40d0ed0 msgr2=0x7f36c4155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c3fff640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36c40d0ed0 0x7f36c4155e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c3fff640 1 -- 172.21.15.46:0/1313620627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4009400 con 0x7f36c406ea50 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c3fff640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 0x7f36c4156510 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00c430 tx=0x7f36ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac0052d0 con 0x7f36c406ea50 2023-12-16T21:14:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ac005450 con 0x7f36c406ea50 2023-12-16T21:14:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c413d490 con 0x7f36c406ea50 2023-12-16T21:14:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ac003830 con 0x7f36c406ea50 2023-12-16T21:14:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.132+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c413da50 con 0x7f36c406ea50 2023-12-16T21:14:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.133+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c4069400 con 0x7f36c406ea50 2023-12-16T21:14:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.134+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f36ac003990 con 0x7f36c406ea50 2023-12-16T21:14:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.134+0000 7f36c0ff9640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36a406ae40 0x7f36a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.134+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f36ac088870 con 0x7f36c406ea50 2023-12-16T21:14:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.134+0000 7f36c37fe640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36a406ae40 0x7f36a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.135+0000 7f36c37fe640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36a406ae40 0x7f36a406d300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f36b4034000 tx=0x7f36b4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.138+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f36ac053d20 con 0x7f36c406ea50 2023-12-16T21:14:34.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.315+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f36c4070d30 con 0x7f36c406ea50 2023-12-16T21:14:34.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.315+0000 7f36c0ff9640 1 -- 172.21.15.46:0/1313620627 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7f36ac017090 con 0x7f36c406ea50 2023-12-16T21:14:34.315 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:34.316 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2023-12-16T21:14:34.316 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:14:34.316 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2023-12-16T21:14:34.316 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2023-12-16T21:14:34.316 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:34.316 INFO:teuthology.orchestra.run.smithi046.stdout:] 2023-12-16T21:14:34.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36a406ae40 msgr2=0x7f36a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36a406ae40 0x7f36a406d300 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f36b4034000 tx=0x7f36b4008030 comp rx=0 tx=0).stop 2023-12-16T21:14:34.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 msgr2=0x7f36c4156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 0x7f36c4156510 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00c430 tx=0x7f36ac00c8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 shutdown_connections 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36a406ae40 0x7f36a406d300 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36c4160fb0 0x7f36c413df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36c40d0ed0 0x7f36c4155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 --2- 172.21.15.46:0/1313620627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36c406ea50 0x7f36c4156510 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 >> 172.21.15.46:0/1313620627 conn(0x7f36c406a6b0 msgr2=0x7f36c415bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 shutdown_connections 2023-12-16T21:14:34.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.317+0000 7f36ca675640 1 -- 172.21.15.46:0/1313620627 wait complete. 2023-12-16T21:14:34.327 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-16T21:14:34.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-16T21:14:34.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.439+0000 7f10dd5f5640 1 Processor -- start 2023-12-16T21:14:34.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.439+0000 7f10dd5f5640 1 -- start start 2023-12-16T21:14:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10dd5f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d806ea50 0x7f10d806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10dd5f5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10d80d0ed0 0x7f10d80d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10dd5f5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d8161d20 con 0x7f10d8160fb0 2023-12-16T21:14:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10dd5f5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d8161b20 con 0x7f10d80d0ed0 2023-12-16T21:14:34.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10dd5f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d806f370 con 0x7f10d806ea50 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10d77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10d80d0ed0 0x7f10d80d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10d6ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d806ea50 0x7f10d806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10d77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10d80d0ed0 0x7f10d80d1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:40332/0 (socket says 172.21.15.46:40332) 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.440+0000 7f10d77fe640 1 -- 172.21.15.46:0/678006441 learned_addr learned my addr 172.21.15.46:0/678006441 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.441+0000 7f10d77fe640 1 -- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d806ea50 msgr2=0x7f10d806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.441+0000 7f10d77fe640 1 --2- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d806ea50 0x7f10d806ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.441+0000 7f10d77fe640 1 -- 172.21.15.46:0/678006441 >> v1:172.21.15.121:6789/0 conn(0x7f10d8160fb0 legacy=0x7f10d8161410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.441+0000 7f10d77fe640 1 -- 172.21.15.46:0/678006441 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d815e760 con 0x7f10d80d0ed0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.441+0000 7f10d77fe640 1 --2- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d80d0ed0 0x7f10d80d1340 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009e50 tx=0x7f10c802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df18813d073b499c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10d5ffb640 1 -- 172.21.15.46:0/678006441 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c802cb00 con 0x7f10d80d0ed0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10d5ffb640 1 -- 172.21.15.46:0/678006441 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c802cc60 con 0x7f10d80d0ed0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10d5ffb640 1 -- 172.21.15.46:0/678006441 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80343f0 con 0x7f10d80d0ed0 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 -- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d80d0ed0 msgr2=0x7f10d80d1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d80d0ed0 0x7f10d80d1340 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f10c8009e50 tx=0x7f10c802bf60 comp rx=0 tx=0).stop 2023-12-16T21:14:34.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 -- 172.21.15.46:0/678006441 shutdown_connections 2023-12-16T21:14:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d80d0ed0 0x7f10d80d1340 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/678006441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d806ea50 0x7f10d806ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 -- 172.21.15.46:0/678006441 >> 172.21.15.46:0/678006441 conn(0x7f10d806a6b0 msgr2=0x7f10d806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 -- 172.21.15.46:0/678006441 shutdown_connections 2023-12-16T21:14:34.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.442+0000 7f10dd5f5640 1 -- 172.21.15.46:0/678006441 wait complete. 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.443+0000 7f10dd5f5640 1 Processor -- start 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 -- start start 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d806ea50 0x7f10d81569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 0x7f10d81f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d8160fb0 0x7f10d815e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8156f20 con 0x7f10d80d0ed0 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d815aeb0 con 0x7f10d806ea50 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10dd5f5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d80b36e0 con 0x7f10d8160fb0 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 0x7f10d81f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10d67fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 0x7f10d81f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41590/0 (socket says 172.21.15.46:41590) 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.444+0000 7f10d67fc640 1 -- 172.21.15.46:0/1029167795 learned_addr learned my addr 172.21.15.46:0/1029167795 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d77fe640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d8160fb0 0x7f10d815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d6ffd640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d806ea50 0x7f10d81569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d67fc640 1 -- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d8160fb0 msgr2=0x7f10d815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d67fc640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d8160fb0 0x7f10d815e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d67fc640 1 -- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d806ea50 msgr2=0x7f10d81569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d67fc640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d806ea50 0x7f10d81569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d67fc640 1 -- 172.21.15.46:0/1029167795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8009b30 con 0x7f10d80d0ed0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10d67fc640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 0x7f10d81f2ea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f10c000c450 tx=0x7f10c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00052d0 con 0x7f10d80d0ed0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0005430 con 0x7f10d80d0ed0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0012870 con 0x7f10d80d0ed0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d80b39c0 con 0x7f10d80d0ed0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.445+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d80b3f80 con 0x7f10d80d0ed0 2023-12-16T21:14:34.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.446+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d8069400 con 0x7f10d80d0ed0 2023-12-16T21:14:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.447+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f10c00129d0 con 0x7f10d80d0ed0 2023-12-16T21:14:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.447+0000 7f10cf7fe640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b806ae40 0x7f10b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.447+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f10c0088d80 con 0x7f10d80d0ed0 2023-12-16T21:14:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.447+0000 7f10d6ffd640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b806ae40 0x7f10b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.448+0000 7f10d6ffd640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b806ae40 0x7f10b806d300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f10d81f63c0 tx=0x7f10d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.451+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f10c0054210 con 0x7f10d80d0ed0 2023-12-16T21:14:34.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.629+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f10d8070d30 con 0x7f10d80d0ed0 2023-12-16T21:14:34.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.728+0000 7f10cf7fe640 1 -- 172.21.15.46:0/1029167795 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f10c0057830 con 0x7f10d80d0ed0 2023-12-16T21:14:34.728 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b806ae40 msgr2=0x7f10b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b806ae40 0x7f10b806d300 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f10d81f63c0 tx=0x7f10d0008040 comp rx=0 tx=0).stop 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 msgr2=0x7f10d81f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 0x7f10d81f2ea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f10c000c450 tx=0x7f10c000c910 comp rx=0 tx=0).stop 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 shutdown_connections 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b806ae40 0x7f10b806d300 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d80d0ed0 0x7f10d81f2ea0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d8160fb0 0x7f10d815e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 --2- 172.21.15.46:0/1029167795 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d806ea50 0x7f10d81569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.731+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 >> 172.21.15.46:0/1029167795 conn(0x7f10d806a6b0 msgr2=0x7f10d80b14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:34.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.732+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 shutdown_connections 2023-12-16T21:14:34.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.732+0000 7f10dd5f5640 1 -- 172.21.15.46:0/1029167795 wait complete. 2023-12-16T21:14:34.745 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-16T21:14:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 Processor -- start 2023-12-16T21:14:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 -- start start 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f10d015d4d0 0x7f10d00d3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d00afe40 0x7f10d00d18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10d00d1de0 0x7f10d00b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d00b1180 con 0x7f10d00d1de0 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d00d2210 con 0x7f10d015d4d0 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.860+0000 7f10d5a9b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00d13d0 con 0x7f10d00afe40 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10d00d1de0 0x7f10d00b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f10d00d1de0 0x7f10d00b2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41596/0 (socket says 172.21.15.46:41596) 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 -- 172.21.15.46:0/294096752 learned_addr learned my addr 172.21.15.46:0/294096752 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10ceffd640 1 --2- 172.21.15.46:0/294096752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d00afe40 0x7f10d00d18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cf7fe640 1 --2- 172.21.15.46:0/294096752 >> v2:172.21.15.121:3300/0 conn(0x7f10d015d4d0 0x7f10d00d3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 -- 172.21.15.46:0/294096752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d00afe40 msgr2=0x7f10d00d18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 --2- 172.21.15.46:0/294096752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d00afe40 0x7f10d00d18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 -- 172.21.15.46:0/294096752 >> v2:172.21.15.121:3300/0 conn(0x7f10d015d4d0 msgr2=0x7f10d00d3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 --2- 172.21.15.46:0/294096752 >> v2:172.21.15.121:3300/0 conn(0x7f10d015d4d0 0x7f10d00d3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.861+0000 7f10cffff640 1 -- 172.21.15.46:0/294096752 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d00b0280 con 0x7f10d00d1de0 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10cffff640 1 --2- 172.21.15.46:0/294096752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00d1de0 0x7f10d00b2b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f10d00b5dc0 tx=0x7f10c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44ada06cb370f17 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10ce7fc640 1 -- 172.21.15.46:0/294096752 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c802c9e0 con 0x7f10d00d1de0 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10ce7fc640 1 -- 172.21.15.46:0/294096752 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c802cb40 con 0x7f10d00d1de0 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10ce7fc640 1 -- 172.21.15.46:0/294096752 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c80343f0 con 0x7f10d00d1de0 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 -- 172.21.15.46:0/294096752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00d1de0 msgr2=0x7f10d00b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/294096752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00d1de0 0x7f10d00b2b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f10d00b5dc0 tx=0x7f10c802be40 comp rx=0 tx=0).stop 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 -- 172.21.15.46:0/294096752 shutdown_connections 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/294096752 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00d1de0 0x7f10d00b2b70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/294096752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d00afe40 0x7f10d00d18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/294096752 >> v2:172.21.15.121:3300/0 conn(0x7f10d015d4d0 0x7f10d00d3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 -- 172.21.15.46:0/294096752 >> 172.21.15.46:0/294096752 conn(0x7f10d0066bf0 msgr2=0x7f10d009c550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.862+0000 7f10d5a9b640 1 -- 172.21.15.46:0/294096752 shutdown_connections 2023-12-16T21:14:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.863+0000 7f10d5a9b640 1 -- 172.21.15.46:0/294096752 wait complete. 2023-12-16T21:14:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.864+0000 7f10d5a9b640 1 Processor -- start 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.864+0000 7f10d5a9b640 1 -- start start 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 0x7f10d00cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d00d1de0 0x7f10d015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d015d4d0 0x7f10d0151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00c82e0 con 0x7f10d00afe40 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00c85c0 con 0x7f10d00d1de0 2023-12-16T21:14:34.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00c88a0 con 0x7f10d015d4d0 2023-12-16T21:14:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 0x7f10d00cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 0x7f10d00cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41598/0 (socket says 172.21.15.46:41598) 2023-12-16T21:14:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 -- 172.21.15.46:0/3307687769 learned_addr learned my addr 172.21.15.46:0/3307687769 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cffff640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d015d4d0 0x7f10d0151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 -- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d015d4d0 msgr2=0x7f10d0151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.867 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d015d4d0 0x7f10d0151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10ceffd640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d00d1de0 0x7f10d015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 -- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d00d1de0 msgr2=0x7f10d015e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:34.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d00d1de0 0x7f10d015e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:34.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 -- 172.21.15.46:0/3307687769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8009b30 con 0x7f10d00afe40 2023-12-16T21:14:34.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10cf7fe640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 0x7f10d00cba90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f10c000e9b0 tx=0x7f10c000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0009670 con 0x7f10d00afe40 2023-12-16T21:14:34.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c000c9a0 con 0x7f10d00afe40 2023-12-16T21:14:34.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d00c8b80 con 0x7f10d00afe40 2023-12-16T21:14:34.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c000cd50 con 0x7f10d00afe40 2023-12-16T21:14:34.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.865+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d00c9380 con 0x7f10d00afe40 2023-12-16T21:14:34.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.866+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d0067420 con 0x7f10d00afe40 2023-12-16T21:14:34.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.867+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f10c0013470 con 0x7f10d00afe40 2023-12-16T21:14:34.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.867+0000 7f10c7fff640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b006ae40 0x7f10b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:34.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.868+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f10c0089d50 con 0x7f10d00afe40 2023-12-16T21:14:34.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.868+0000 7f10ceffd640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b006ae40 0x7f10b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:34.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.868+0000 7f10ceffd640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b006ae40 0x7f10b006d300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f10d00c9940 tx=0x7f10b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:34.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:34.871+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f10c0055150 con 0x7f10d00afe40 2023-12-16T21:14:35.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.047+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f10d00d0a50 con 0x7f10d00afe40 2023-12-16T21:14:35.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.730+0000 7f10c7fff640 1 -- 172.21.15.46:0/3307687769 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f10c0058770 con 0x7f10d00afe40 2023-12-16T21:14:35.730 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b006ae40 msgr2=0x7f10b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b006ae40 0x7f10b006d300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f10d00c9940 tx=0x7f10b8008040 comp rx=0 tx=0).stop 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 msgr2=0x7f10d00cba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 0x7f10d00cba90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f10c000e9b0 tx=0x7f10c000ee70 comp rx=0 tx=0).stop 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 shutdown_connections 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f10b006ae40 0x7f10b006d300 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10d015d4d0 0x7f10d0151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10d00d1de0 0x7f10d015e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 --2- 172.21.15.46:0/3307687769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10d00afe40 0x7f10d00cba90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 >> 172.21.15.46:0/3307687769 conn(0x7f10d0066bf0 msgr2=0x7f10d00ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 shutdown_connections 2023-12-16T21:14:35.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.734+0000 7f10d5a9b640 1 -- 172.21.15.46:0/3307687769 wait complete. 2023-12-16T21:14:35.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-16T21:14:35.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.872+0000 7f41ba689640 1 Processor -- start 2023-12-16T21:14:35.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.872+0000 7f41ba689640 1 -- start start 2023-12-16T21:14:35.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41ba689640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f41b4160fb0 0x7f41b4161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41ba689640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 0x7f41b40d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41ba689640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b4161d20 con 0x7f41b406ea50 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41ba689640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41b415e760 con 0x7f41b4160fb0 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41ba689640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b415d9a0 con 0x7f41b40d0ed0 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b3fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f41b406ea50 0x7f41b406ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36556/0 (socket says 172.21.15.46:36556) 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b3fff640 1 -- 172.21.15.46:0/4247855635 learned_addr learned my addr 172.21.15.46:0/4247855635 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b8c26640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 0x7f41b40d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b37fe640 1 --2- 172.21.15.46:0/4247855635 >> v2:172.21.15.121:3300/0 conn(0x7f41b4160fb0 0x7f41b4161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909866872 0 0) 0x7f41b4161d20 con 0x7f41b406ea50 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a0003520 con 0x7f41b406ea50 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4284524478 0 0) 0x7f41a0003520 con 0x7f41b406ea50 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.873+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 msgr2=0x7f41b40d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 0x7f41b40d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 >> v2:172.21.15.121:3300/0 conn(0x7f41b4160fb0 msgr2=0x7f41b4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 --2- 172.21.15.46:0/4247855635 >> v2:172.21.15.121:3300/0 conn(0x7f41b4160fb0 0x7f41b4161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b415dc80 con 0x7f41b406ea50 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f419c001670 con 0x7f41b406ea50 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 >> v1:172.21.15.46:6789/0 conn(0x7f41b406ea50 legacy=0x7f41b406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 0x7f41a0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b415e760 con 0x7f41a0003ef0 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b415d9a0 con 0x7f41a0004d00 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a000b980 con 0x7f41b40d0ed0 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b8c26640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b37fe640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b3fff640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 0x7f41a0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b37fe640 1 -- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 msgr2=0x7f41a0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b37fe640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b40d0ed0 0x7f41a0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b37fe640 1 -- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41a0004d00 msgr2=0x7f41a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b37fe640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41a0004d00 0x7f41a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.874+0000 7f41b37fe640 1 -- 172.21.15.46:0/4247855635 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0003520 con 0x7f41a0003ef0 2023-12-16T21:14:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41b37fe640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41a00047c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009ac0 tx=0x7f41a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4923bc926e07d39 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a402c6d0 con 0x7f41a0003ef0 2023-12-16T21:14:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a402c830 con 0x7f41a0003ef0 2023-12-16T21:14:35.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41b2ffd640 1 -- 172.21.15.46:0/4247855635 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a402cbc0 con 0x7f41a0003ef0 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41ba689640 1 -- 172.21.15.46:0/4247855635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 msgr2=0x7f41a00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41ba689640 1 --2- 172.21.15.46:0/4247855635 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41a00047c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f41a4009ac0 tx=0x7f41a00057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.875+0000 7f41b8c26640 1 -- 172.21.15.46:0/4247855635 reap_dead start 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.876+0000 7f41ba689640 1 -- 172.21.15.46:0/4247855635 shutdown_connections 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.876+0000 7f41ba689640 1 -- 172.21.15.46:0/4247855635 >> 172.21.15.46:0/4247855635 conn(0x7f41b406a6b0 msgr2=0x7f41b406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.876+0000 7f41ba689640 1 -- 172.21.15.46:0/4247855635 shutdown_connections 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.876+0000 7f41ba689640 1 -- 172.21.15.46:0/4247855635 wait complete. 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.877+0000 7f41ba689640 1 Processor -- start 2023-12-16T21:14:35.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.877+0000 7f41ba689640 1 -- start start 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41ba689640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41b413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41ba689640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b4156500 0x7f41b4156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41ba689640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b4159520 0x7f41b41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41ba689640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b413e000 con 0x7f41a0003ef0 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41ba689640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4156ec0 con 0x7f41b4156500 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41ba689640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41b4154f00 con 0x7f41b4159520 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41b413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41b413dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41608/0 (socket says 172.21.15.46:41608) 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 -- 172.21.15.46:0/881550454 learned_addr learned my addr 172.21.15.46:0/881550454 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b8c26640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b4159520 0x7f41b41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b37fe640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b4156500 0x7f41b4156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 -- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b4159520 msgr2=0x7f41b41599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b4159520 0x7f41b41599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 -- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b4156500 msgr2=0x7f41b4156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b4156500 0x7f41b4156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 -- 172.21.15.46:0/881550454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b41551e0 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b8c26640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b4159520 0x7f41b41599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b3fff640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41b413dac0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f419c007340 tx=0x7f419c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.878+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00f860 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.879+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c00f9c0 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.879+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b41554c0 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.879+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00fd70 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.879+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b4155a80 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.880+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41b4066e90 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.880+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f419c01d460 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.880+0000 7f41b0ff9640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f419406ae70 0x7f419406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.881+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f419c089730 con 0x7f41a0003ef0 2023-12-16T21:14:35.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.881+0000 7f41b37fe640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f419406ae70 0x7f419406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:35.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.881+0000 7f41b37fe640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f419406ae70 0x7f419406d330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f41a4036000 tx=0x7f41a4030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:35.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:35.885+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f419c054900 con 0x7f41a0003ef0 2023-12-16T21:14:36.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.096+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f41b4069400 con 0x7f41a0003ef0 2023-12-16T21:14:36.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.745+0000 7f41b0ff9640 1 -- 172.21.15.46:0/881550454 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) v1 ==== 172+0+0 (secure 0 0 0) 0x7f419c057f20 con 0x7f41a0003ef0 2023-12-16T21:14:36.745 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 4 and data pool 5 2023-12-16T21:14:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f419406ae70 msgr2=0x7f419406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f419406ae70 0x7f419406d330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f41a4036000 tx=0x7f41a4030000 comp rx=0 tx=0).stop 2023-12-16T21:14:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 msgr2=0x7f41b413dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41b413dac0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f419c007340 tx=0x7f419c007800 comp rx=0 tx=0).stop 2023-12-16T21:14:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 shutdown_connections 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f419406ae70 0x7f419406d330 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41b4159520 0x7f41b41599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41b4156500 0x7f41b4156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 --2- 172.21.15.46:0/881550454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f41a0003ef0 0x7f41b413dac0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.748+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 >> 172.21.15.46:0/881550454 conn(0x7f41b406a6b0 msgr2=0x7f41b415c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.749+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 shutdown_connections 2023-12-16T21:14:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.749+0000 7f41ba689640 1 -- 172.21.15.46:0/881550454 wait complete. 2023-12-16T21:14:36.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.873+0000 7f6bc36cf640 1 Processor -- start 2023-12-16T21:14:36.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.873+0000 7f6bc36cf640 1 -- start start 2023-12-16T21:14:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc36cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6bbc0d2910 0x7f6bbc0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc36cf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6bbc0d3730 0x7f6bbc0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:36.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc36cf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc0b2b10 con 0x7f6bbc15d330 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc36cf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc0d3bc0 con 0x7f6bbc0d2910 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc36cf640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc0d1590 con 0x7f6bbc0d3730 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc0c6a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6bbc0d2910 0x7f6bbc0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc0c6a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6bbc0d2910 0x7f6bbc0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41610/0 (socket says 172.21.15.46:41610) 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc0c6a640 1 -- 172.21.15.46:0/3562030618 learned_addr learned my addr 172.21.15.46:0/3562030618 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.874+0000 7f6bc146b640 1 --1- 172.21.15.46:0/3562030618 >> v1:172.21.15.46:6789/0 conn(0x7f6bbc15d330 0x7f6bbc0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36572/0 (socket says 172.21.15.46:36572) 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.875+0000 7f6bc1c6c640 1 --2- 172.21.15.46:0/3562030618 >> v2:172.21.15.121:3300/0 conn(0x7f6bbc0d3730 0x7f6bbc0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.875+0000 7f6bc0c6a640 1 -- 172.21.15.46:0/3562030618 >> v2:172.21.15.121:3300/0 conn(0x7f6bbc0d3730 msgr2=0x7f6bbc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.875+0000 7f6bc0c6a640 1 --2- 172.21.15.46:0/3562030618 >> v2:172.21.15.121:3300/0 conn(0x7f6bbc0d3730 0x7f6bbc0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.875+0000 7f6bc0c6a640 1 -- 172.21.15.46:0/3562030618 >> v1:172.21.15.46:6789/0 conn(0x7f6bbc15d330 legacy=0x7f6bbc0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:36.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.875+0000 7f6bc0c6a640 1 -- 172.21.15.46:0/3562030618 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0d1870 con 0x7f6bbc0d2910 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.875+0000 7f6bc0c6a640 1 --2- 172.21.15.46:0/3562030618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0d31f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc06c500 tx=0x7f6bb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d5cdf9c636d860 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bbbfff640 1 -- 172.21.15.46:0/3562030618 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0031870 con 0x7f6bbc0d2910 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bbbfff640 1 -- 172.21.15.46:0/3562030618 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb00319d0 con 0x7f6bbc0d2910 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bbbfff640 1 -- 172.21.15.46:0/3562030618 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0031d60 con 0x7f6bbc0d2910 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 -- 172.21.15.46:0/3562030618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc0d2910 msgr2=0x7f6bbc0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/3562030618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0d31f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc06c500 tx=0x7f6bb002b910 comp rx=0 tx=0).stop 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 -- 172.21.15.46:0/3562030618 shutdown_connections 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/3562030618 >> v2:172.21.15.121:3300/0 conn(0x7f6bbc0d3730 0x7f6bbc0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/3562030618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0d31f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 -- 172.21.15.46:0/3562030618 >> 172.21.15.46:0/3562030618 conn(0x7f6bbc072020 msgr2=0x7f6bbc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 -- 172.21.15.46:0/3562030618 shutdown_connections 2023-12-16T21:14:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.876+0000 7f6bc36cf640 1 -- 172.21.15.46:0/3562030618 wait complete. 2023-12-16T21:14:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.877+0000 7f6bc36cf640 1 Processor -- start 2023-12-16T21:14:36.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 -- start start 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bbc0d3730 0x7f6bbc0c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 0x7f6bbc0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0cb840 con 0x7f6bbc15d330 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0cabc0 con 0x7f6bbc0d2910 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc36cf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc0caea0 con 0x7f6bbc0d3730 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc1c6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 0x7f6bbc0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc1c6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 0x7f6bbc0c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41626/0 (socket says 172.21.15.46:41626) 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.878+0000 7f6bc1c6c640 1 -- 172.21.15.46:0/148138238 learned_addr learned my addr 172.21.15.46:0/148138238 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc0c6a640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bbc0d3730 0x7f6bbc0c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc146b640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc1c6c640 1 -- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bbc0d3730 msgr2=0x7f6bbc0c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:36.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc1c6c640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bbc0d3730 0x7f6bbc0c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc1c6c640 1 -- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bbc0d2910 msgr2=0x7f6bbc0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc1c6c640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc1c6c640 1 -- 172.21.15.46:0/148138238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00094d0 con 0x7f6bbc15d330 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc146b640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc1c6c640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 0x7f6bbc0c6ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6bb400d800 tx=0x7f6bb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb400b6c0 con 0x7f6bbc15d330 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb400b840 con 0x7f6bbc15d330 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc0cb180 con 0x7f6bbc15d330 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb400bbf0 con 0x7f6bbc15d330 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.879+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc0cbc60 con 0x7f6bbc15d330 2023-12-16T21:14:36.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.880+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc15f0c0 con 0x7f6bbc15d330 2023-12-16T21:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.881+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6bb400bd50 con 0x7f6bbc15d330 2023-12-16T21:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.881+0000 7f6bb9ffb640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ba806ae40 0x7f6ba806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.881+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bb4089b90 con 0x7f6bbc15d330 2023-12-16T21:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.881+0000 7f6bc146b640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ba806ae40 0x7f6ba806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:36.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.882+0000 7f6bc146b640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ba806ae40 0x7f6ba806d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6bac007fd0 tx=0x7f6bac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:36.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:36.886+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6bb4054c50 con 0x7f6bbc15d330 2023-12-16T21:14:37.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.064+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bbc069400 con 0x7f6bbc15d330 2023-12-16T21:14:37.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.065+0000 7f6bb9ffb640 1 -- 172.21.15.46:0/148138238 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f6bb4058270 con 0x7f6bbc15d330 2023-12-16T21:14:37.066 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:37.066 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":33,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:36.097709+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ba806ae40 msgr2=0x7f6ba806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ba806ae40 0x7f6ba806d300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6bac007fd0 tx=0x7f6bac009040 comp rx=0 tx=0).stop 2023-12-16T21:14:37.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 msgr2=0x7f6bbc0c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 0x7f6bbc0c6ba0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6bb400d800 tx=0x7f6bb400dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 shutdown_connections 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ba806ae40 0x7f6ba806d300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bbc15d330 0x7f6bbc0c6ba0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6bbc0d3730 0x7f6bbc0c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 --2- 172.21.15.46:0/148138238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bbc0d2910 0x7f6bbc0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 >> 172.21.15.46:0/148138238 conn(0x7f6bbc072020 msgr2=0x7f6bbc072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.067+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 shutdown_connections 2023-12-16T21:14:37.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.068+0000 7f6bc36cf640 1 -- 172.21.15.46:0/148138238 wait complete. 2023-12-16T21:14:37.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:37.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.185+0000 7f5223cfc640 1 Processor -- start 2023-12-16T21:14:37.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.185+0000 7f5223cfc640 1 -- start start 2023-12-16T21:14:37.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5223cfc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f521c143490 0x7f521c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5223cfc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f521c15abf0 0x7f521c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5223cfc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c15e2a0 con 0x7f521c15e730 2023-12-16T21:14:37.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5223cfc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c15b520 con 0x7f521c15abf0 2023-12-16T21:14:37.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5223cfc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521c143870 con 0x7f521c143490 2023-12-16T21:14:37.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5222299640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f521c15abf0 0x7f521c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5221297640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f521c15e730 0x7f521c15a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36586/0 (socket says 172.21.15.46:36586) 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5222299640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f521c15abf0 0x7f521c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41642/0 (socket says 172.21.15.46:41642) 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5221297640 1 -- 172.21.15.46:0/2195945067 learned_addr learned my addr 172.21.15.46:0/2195945067 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.186+0000 7f5221a98640 1 --2- 172.21.15.46:0/2195945067 >> v2:172.21.15.121:3300/0 conn(0x7f521c143490 0x7f521c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5222299640 1 -- 172.21.15.46:0/2195945067 >> v2:172.21.15.121:3300/0 conn(0x7f521c143490 msgr2=0x7f521c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5222299640 1 --2- 172.21.15.46:0/2195945067 >> v2:172.21.15.121:3300/0 conn(0x7f521c143490 0x7f521c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5222299640 1 -- 172.21.15.46:0/2195945067 >> v1:172.21.15.46:6789/0 conn(0x7f521c15e730 legacy=0x7f521c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5222299640 1 -- 172.21.15.46:0/2195945067 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c15b240 con 0x7f521c15abf0 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5220a96640 1 -- 172.21.15.46:0/2195945067 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456602030 0 0) 0x7f521c15e2a0 con 0x7f521c15e730 2023-12-16T21:14:37.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5221a98640 1 --2- 172.21.15.46:0/2195945067 >> v2:172.21.15.121:3300/0 conn(0x7f521c143490 0x7f521c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:37.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.187+0000 7f5222299640 1 --2- 172.21.15.46:0/2195945067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c15abf0 0x7f521c0b1230 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f521c06e620 tx=0x7f520c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16875465c34ef03f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5220a96640 1 -- 172.21.15.46:0/2195945067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c02cc80 con 0x7f521c15abf0 2023-12-16T21:14:37.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5220a96640 1 -- 172.21.15.46:0/2195945067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520c02cde0 con 0x7f521c15abf0 2023-12-16T21:14:37.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5220a96640 1 -- 172.21.15.46:0/2195945067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c034660 con 0x7f521c15abf0 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 -- 172.21.15.46:0/2195945067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c15abf0 msgr2=0x7f521c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 --2- 172.21.15.46:0/2195945067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c15abf0 0x7f521c0b1230 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f521c06e620 tx=0x7f520c02bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 -- 172.21.15.46:0/2195945067 shutdown_connections 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 --2- 172.21.15.46:0/2195945067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c15abf0 0x7f521c0b1230 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 --2- 172.21.15.46:0/2195945067 >> v2:172.21.15.121:3300/0 conn(0x7f521c143490 0x7f521c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 -- 172.21.15.46:0/2195945067 >> 172.21.15.46:0/2195945067 conn(0x7f521c06a4d0 msgr2=0x7f521c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 -- 172.21.15.46:0/2195945067 shutdown_connections 2023-12-16T21:14:37.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.188+0000 7f5223cfc640 1 -- 172.21.15.46:0/2195945067 wait complete. 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.189+0000 7f5223cfc640 1 Processor -- start 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.189+0000 7f5223cfc640 1 -- start start 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5223cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 0x7f521c155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5223cfc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f521c15abf0 0x7f521c0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5223cfc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521c15e730 0x7f521c140160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5223cfc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c13cc70 con 0x7f521c143490 2023-12-16T21:14:37.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5223cfc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c13eb40 con 0x7f521c15e730 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5223cfc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c13ed70 con 0x7f521c15abf0 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5221a98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 0x7f521c155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5221a98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 0x7f521c155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41658/0 (socket says 172.21.15.46:41658) 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5221a98640 1 -- 172.21.15.46:0/1157598911 learned_addr learned my addr 172.21.15.46:0/1157598911 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5222299640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521c15e730 0x7f521c140160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.190+0000 7f5221297640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f521c15abf0 0x7f521c0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5221a98640 1 -- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f521c15abf0 msgr2=0x7f521c0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5221a98640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f521c15abf0 0x7f521c0ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5221a98640 1 -- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521c15e730 msgr2=0x7f521c140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5221a98640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521c15e730 0x7f521c140160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5221a98640 1 -- 172.21.15.46:0/1157598911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520c009a70 con 0x7f521c143490 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5221a98640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 0x7f521c155cd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f521400e730 tx=0x7f521400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214004230 con 0x7f521c143490 2023-12-16T21:14:37.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5214004390 con 0x7f521c143490 2023-12-16T21:14:37.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c13f050 con 0x7f521c143490 2023-12-16T21:14:37.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400c8f0 con 0x7f521c143490 2023-12-16T21:14:37.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.191+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c13fc40 con 0x7f521c143490 2023-12-16T21:14:37.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.192+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c069400 con 0x7f521c143490 2023-12-16T21:14:37.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.192+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f521400ca50 con 0x7f521c143490 2023-12-16T21:14:37.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.193+0000 7f521a7fc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f520806ae40 0x7f520806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.193+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5214089190 con 0x7f521c143490 2023-12-16T21:14:37.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.193+0000 7f5221297640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f520806ae40 0x7f520806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.193+0000 7f5221297640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f520806ae40 0x7f520806d300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f521c141800 tx=0x7f5210007c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.197+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f52140542d0 con 0x7f521c143490 2023-12-16T21:14:37.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.375+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f521c0b2820 con 0x7f521c143490 2023-12-16T21:14:37.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.376+0000 7f521a7fc640 1 -- 172.21.15.46:0/1157598911 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f52140578f0 con 0x7f521c143490 2023-12-16T21:14:37.377 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:37.377 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":33,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:36.097709+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:37.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.378+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f520806ae40 msgr2=0x7f520806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.378+0000 7f5223cfc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f520806ae40 0x7f520806d300 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f521c141800 tx=0x7f5210007c00 comp rx=0 tx=0).stop 2023-12-16T21:14:37.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.378+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 msgr2=0x7f521c155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.378+0000 7f5223cfc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 0x7f521c155cd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f521400e730 tx=0x7f521400ebf0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 shutdown_connections 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f520806ae40 0x7f520806d300 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521c15e730 0x7f521c140160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f521c15abf0 0x7f521c0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 --2- 172.21.15.46:0/1157598911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f521c143490 0x7f521c155cd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 >> 172.21.15.46:0/1157598911 conn(0x7f521c06a4d0 msgr2=0x7f521c0717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 shutdown_connections 2023-12-16T21:14:37.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.379+0000 7f5223cfc640 1 -- 172.21.15.46:0/1157598911 wait complete. 2023-12-16T21:14:37.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-16T21:14:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 Processor -- start 2023-12-16T21:14:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 -- start start 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96c415d330 0x7f96c40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c40d2910 0x7f96c40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c40b2b10 con 0x7f96c40d3730 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c40d3b50 con 0x7f96c415d330 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.501+0000 7f96ca21e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c40d1590 con 0x7f96c40d2910 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96c415d330 0x7f96c40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96c415d330 0x7f96c40b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41670/0 (socket says 172.21.15.46:41670) 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 -- 172.21.15.46:0/3380014124 learned_addr learned my addr 172.21.15.46:0/3380014124 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c2ffd640 1 --2- 172.21.15.46:0/3380014124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c40d2910 0x7f96c40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 -- 172.21.15.46:0/3380014124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c40d2910 msgr2=0x7f96c40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 --2- 172.21.15.46:0/3380014124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c40d2910 0x7f96c40d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 -- 172.21.15.46:0/3380014124 >> v1:172.21.15.121:6789/0 conn(0x7f96c40d3730 legacy=0x7f96c40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.502+0000 7f96c37fe640 1 -- 172.21.15.46:0/3380014124 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c40d1870 con 0x7f96c415d330 2023-12-16T21:14:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96c37fe640 1 --2- 172.21.15.46:0/3380014124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c415d330 0x7f96c40b8b80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009870 tx=0x7f96b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bbd5ac90a32cfdc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96c27fc640 1 -- 172.21.15.46:0/3380014124 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4031640 con 0x7f96c415d330 2023-12-16T21:14:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96c27fc640 1 -- 172.21.15.46:0/3380014124 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b40317a0 con 0x7f96c415d330 2023-12-16T21:14:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96c27fc640 1 -- 172.21.15.46:0/3380014124 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4031b30 con 0x7f96c415d330 2023-12-16T21:14:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 -- 172.21.15.46:0/3380014124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c415d330 msgr2=0x7f96c40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 --2- 172.21.15.46:0/3380014124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c415d330 0x7f96c40b8b80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f96b4009870 tx=0x7f96b402b8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 -- 172.21.15.46:0/3380014124 shutdown_connections 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 --2- 172.21.15.46:0/3380014124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c40d2910 0x7f96c40d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 --2- 172.21.15.46:0/3380014124 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c415d330 0x7f96c40b8b80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 -- 172.21.15.46:0/3380014124 >> 172.21.15.46:0/3380014124 conn(0x7f96c4072020 msgr2=0x7f96c4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.503+0000 7f96ca21e640 1 -- 172.21.15.46:0/3380014124 shutdown_connections 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.504+0000 7f96ca21e640 1 -- 172.21.15.46:0/3380014124 wait complete. 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.505+0000 7f96ca21e640 1 Processor -- start 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.505+0000 7f96ca21e640 1 -- start start 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 0x7f96c415d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c40d3730 0x7f96c414a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c415d330 0x7f96c40c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c414ac20 con 0x7f96c40d2910 2023-12-16T21:14:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c40c7c10 con 0x7f96c40d3730 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c415dda0 con 0x7f96c415d330 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 0x7f96c415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 0x7f96c415d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41674/0 (socket says 172.21.15.46:41674) 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 -- 172.21.15.46:0/1979909122 learned_addr learned my addr 172.21.15.46:0/1979909122 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c3fff640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c415d330 0x7f96c40c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c2ffd640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c40d3730 0x7f96c414a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 -- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c415d330 msgr2=0x7f96c40c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c415d330 0x7f96c40c76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 -- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c40d3730 msgr2=0x7f96c414a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c40d3730 0x7f96c414a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 -- 172.21.15.46:0/1979909122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40094d0 con 0x7f96c40d2910 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96c37fe640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 0x7f96c415d710 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f96b4000c00 tx=0x7f96b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b403ac60 con 0x7f96c40d2910 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b403adc0 con 0x7f96c40d2910 2023-12-16T21:14:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c415e080 con 0x7f96c40d2910 2023-12-16T21:14:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.506+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4044630 con 0x7f96c40d2910 2023-12-16T21:14:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.507+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c415e640 con 0x7f96c40d2910 2023-12-16T21:14:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.508+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c4160970 con 0x7f96c40d2910 2023-12-16T21:14:37.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.508+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f96b402f050 con 0x7f96c40d2910 2023-12-16T21:14:37.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.508+0000 7f96abfff640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96b006ae40 0x7f96b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.508+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96b40ab370 con 0x7f96c40d2910 2023-12-16T21:14:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.508+0000 7f96c2ffd640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96b006ae40 0x7f96b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.509+0000 7f96c2ffd640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96b006ae40 0x7f96b006d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f96c41f65d0 tx=0x7f96b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.512+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f96b40765f0 con 0x7f96c40d2910 2023-12-16T21:14:37.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.723+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f96c4069400 con 0x7f96c40d2910 2023-12-16T21:14:37.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.751+0000 7f96abfff640 1 -- 172.21.15.46:0/1979909122 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v12) v1 ==== 118+0+0 (secure 0 0 0) 0x7f96b4031e60 con 0x7f96c40d2910 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96b006ae40 msgr2=0x7f96b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96b006ae40 0x7f96b006d300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f96c41f65d0 tx=0x7f96b8008040 comp rx=0 tx=0).stop 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 msgr2=0x7f96c415d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 0x7f96c415d710 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f96b4000c00 tx=0x7f96b40046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 shutdown_connections 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96b006ae40 0x7f96b006d300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96c415d330 0x7f96c40c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96c40d3730 0x7f96c414a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 --2- 172.21.15.46:0/1979909122 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96c40d2910 0x7f96c415d710 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 >> 172.21.15.46:0/1979909122 conn(0x7f96c4072020 msgr2=0x7f96c40b7070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 shutdown_connections 2023-12-16T21:14:37.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.753+0000 7f96ca21e640 1 -- 172.21.15.46:0/1979909122 wait complete. 2023-12-16T21:14:37.764 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-16T21:14:37.765 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-16T21:14:37.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.874+0000 7f1d4ada9640 1 Processor -- start 2023-12-16T21:14:37.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.874+0000 7f1d4ada9640 1 -- start start 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d4ada9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d4406f210 0x7f1d44140800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d4ada9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4415ba40 con 0x7f1d44140d40 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d4ada9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d44141190 con 0x7f1d4415c2f0 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d4ada9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4415c6d0 con 0x7f1d4406f210 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d49346640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1d44140d40 0x7f1d4413f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36598/0 (socket says 172.21.15.46:36598) 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d49346640 1 -- 172.21.15.46:0/4202877266 learned_addr learned my addr 172.21.15.46:0/4202877266 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643610768 0 0) 0x7f1d4415ba40 con 0x7f1d44140d40 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c003520 con 0x7f1d44140d40 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d43fff640 1 --2- 172.21.15.46:0/4202877266 >> v2:172.21.15.121:3300/0 conn(0x7f1d4406f210 0x7f1d44140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2771935857 0 0) 0x7f1d2c003520 con 0x7f1d44140d40 2023-12-16T21:14:37.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 >> v2:172.21.15.121:3300/0 conn(0x7f1d4406f210 msgr2=0x7f1d44140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 --2- 172.21.15.46:0/4202877266 >> v2:172.21.15.121:3300/0 conn(0x7f1d4406f210 0x7f1d44140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 >> v1:172.21.15.121:6789/0 conn(0x7f1d4415c2f0 legacy=0x7f1d4406eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d441416e0 con 0x7f1d44140d40 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f1d3c0019f0 con 0x7f1d44140d40 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 >> v1:172.21.15.46:6789/0 conn(0x7f1d44140d40 legacy=0x7f1d4413f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c003970 0x7f1d2c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.875+0000 7f1d437fe640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d2c005180 0x7f1d2c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c00c050 con 0x7f1d2c003970 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c00c190 con 0x7f1d2c005180 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2c00c350 con 0x7f1d2c0047c0 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d48b45640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c003970 0x7f1d2c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d43fff640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d49346640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d2c005180 0x7f1d2c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d48b45640 1 -- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c0047c0 msgr2=0x7f1d2c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d48b45640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d48b45640 1 -- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d2c005180 msgr2=0x7f1d2c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d48b45640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d2c005180 0x7f1d2c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d48b45640 1 -- 172.21.15.46:0/4202877266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c003520 con 0x7f1d2c003970 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d49346640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d2c005180 0x7f1d2c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d43fff640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c0047c0 0x7f1d2c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.876+0000 7f1d49346640 1 -- 172.21.15.46:0/4202877266 reap_dead start 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d48b45640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c003970 0x7f1d2c003db0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1d34001750 tx=0x7f1d2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b414a0209c8246 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d340052d0 con 0x7f1d2c003970 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3402e050 con 0x7f1d2c003970 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d437fe640 1 -- 172.21.15.46:0/4202877266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d340035f0 con 0x7f1d2c003970 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 -- 172.21.15.46:0/4202877266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c003970 msgr2=0x7f1d2c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c003970 0x7f1d2c003db0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1d34001750 tx=0x7f1d2c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 -- 172.21.15.46:0/4202877266 shutdown_connections 2023-12-16T21:14:37.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/4202877266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d2c003970 0x7f1d2c003db0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 -- 172.21.15.46:0/4202877266 >> 172.21.15.46:0/4202877266 conn(0x7f1d4406a6b0 msgr2=0x7f1d4406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:37.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 -- 172.21.15.46:0/4202877266 shutdown_connections 2023-12-16T21:14:37.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.877+0000 7f1d4ada9640 1 -- 172.21.15.46:0/4202877266 wait complete. 2023-12-16T21:14:37.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.878+0000 7f1d4ada9640 1 Processor -- start 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 -- start start 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c003970 0x7f1d440d1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d440b0d90 0x7f1d4415a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 0x7f1d44131560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4415aa80 con 0x7f1d4415a600 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440d17a0 con 0x7f1d440b0d90 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d4ada9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440cfca0 con 0x7f1d2c003970 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d49346640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 0x7f1d44131560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.879+0000 7f1d49346640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 0x7f1d44131560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41692/0 (socket says 172.21.15.46:41692) 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 -- 172.21.15.46:0/881438582 learned_addr learned my addr 172.21.15.46:0/881438582 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d48b45640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c003970 0x7f1d440d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d43fff640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d440b0d90 0x7f1d4415a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 -- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c003970 msgr2=0x7f1d440d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c003970 0x7f1d440d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 -- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d440b0d90 msgr2=0x7f1d4415a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d440b0d90 0x7f1d4415a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 -- 172.21.15.46:0/881438582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440cff80 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d49346640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 0x7f1d44131560 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1d440b4180 tx=0x7f1d3c00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00b860 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c00b9e0 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c00bd90 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d440d0260 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.880+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d440d0820 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.881+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d44069400 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.882+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1d3c007110 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.882+0000 7f1d417fa640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d3006ae70 0x7f1d3006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.882+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d3c089810 con 0x7f1d4415a600 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.882+0000 7f1d48b45640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d3006ae70 0x7f1d3006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:37.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.883+0000 7f1d48b45640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d3006ae70 0x7f1d3006d330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1d34001750 tx=0x7f1d34009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:37.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:37.886+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1d3c054970 con 0x7f1d4415a600 2023-12-16T21:14:38.101 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.100+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1d44070d30 con 0x7f1d4415a600 2023-12-16T21:14:38.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.755+0000 7f1d417fa640 1 -- 172.21.15.46:0/881438582 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v13) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1d3c057f90 con 0x7f1d4415a600 2023-12-16T21:14:38.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.757+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d3006ae70 msgr2=0x7f1d3006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d3006ae70 0x7f1d3006d330 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1d34001750 tx=0x7f1d34009030 comp rx=0 tx=0).stop 2023-12-16T21:14:38.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 msgr2=0x7f1d44131560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 0x7f1d44131560 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1d440b4180 tx=0x7f1d3c00df50 comp rx=0 tx=0).stop 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 shutdown_connections 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d3006ae70 0x7f1d3006d330 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d4415a600 0x7f1d44131560 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d440b0d90 0x7f1d4415a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 --2- 172.21.15.46:0/881438582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2c003970 0x7f1d440d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 >> 172.21.15.46:0/881438582 conn(0x7f1d4406a6b0 msgr2=0x7f1d44153e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 shutdown_connections 2023-12-16T21:14:38.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.758+0000 7f1d4ada9640 1 -- 172.21.15.46:0/881438582 wait complete. 2023-12-16T21:14:38.771 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-16T21:14:38.771 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:38.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.883+0000 7f6d66b61640 1 Processor -- start 2023-12-16T21:14:38.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.883+0000 7f6d66b61640 1 -- start start 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d66b61640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6d600b4370 0x7f6d600b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d66b61640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d60071770 con 0x7f6d600ceb70 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d66b61640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d600ccb60 con 0x7f6d60144980 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d66b61640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6015ced0 con 0x7f6d600b4370 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d5ffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6d600ceb70 0x7f6d600cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36602/0 (socket says 172.21.15.46:36602) 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d5ffff640 1 -- 172.21.15.46:0/1205531859 learned_addr learned my addr 172.21.15.46:0/1205531859 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.884+0000 7f6d650fe640 1 --2- 172.21.15.46:0/1205531859 >> v2:172.21.15.121:3300/0 conn(0x7f6d600b4370 0x7f6d600b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825207220 0 0) 0x7f6d60071770 con 0x7f6d600ceb70 2023-12-16T21:14:38.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d48003520 con 0x7f6d600ceb70 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896464552 0 0) 0x7f6d600ccb60 con 0x7f6d60144980 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d60071770 con 0x7f6d60144980 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 50143544 0 0) 0x7f6d48003520 con 0x7f6d600ceb70 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 >> v2:172.21.15.121:3300/0 conn(0x7f6d600b4370 msgr2=0x7f6d600b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 --2- 172.21.15.46:0/1205531859 >> v2:172.21.15.121:3300/0 conn(0x7f6d600b4370 0x7f6d600b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 >> v1:172.21.15.121:6789/0 conn(0x7f6d60144980 legacy=0x7f6d6015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d600b4ce0 con 0x7f6d600ceb70 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f6d54001670 con 0x7f6d600ceb70 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d650fe640 1 --2- 172.21.15.46:0/1205531859 >> v2:172.21.15.121:3300/0 conn(0x7f6d600b4370 0x7f6d600b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 >> v1:172.21.15.46:6789/0 conn(0x7f6d600ceb70 legacy=0x7f6d600cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d48003b00 0x7f6d48003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d480047c0 0x7f6d48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d48005180 0x7f6d4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6015ced0 con 0x7f6d480047c0 2023-12-16T21:14:38.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4800c020 con 0x7f6d48003b00 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4800c1e0 con 0x7f6d48005180 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d480047c0 0x7f6d48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d648fd640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d48003b00 0x7f6d48003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d650fe640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d48005180 0x7f6d4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5ffff640 1 -- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d48005180 msgr2=0x7f6d4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d48005180 0x7f6d4800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5ffff640 1 -- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d48003b00 msgr2=0x7f6d48003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d48003b00 0x7f6d48003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d650fe640 1 -- 172.21.15.46:0/1205531859 reap_dead start 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.885+0000 7f6d5ffff640 1 -- 172.21.15.46:0/1205531859 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d48003520 con 0x7f6d480047c0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.886+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d480047c0 0x7f6d48004c40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400c190 tx=0x7f6d480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2fefa9f77f77986 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.886+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54005ad0 con 0x7f6d480047c0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.886+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d54005c30 con 0x7f6d480047c0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.886+0000 7f6d5f7fe640 1 -- 172.21.15.46:0/1205531859 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5402c7e0 con 0x7f6d480047c0 2023-12-16T21:14:38.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 -- 172.21.15.46:0/1205531859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d480047c0 msgr2=0x7f6d48004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d480047c0 0x7f6d48004c40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400c190 tx=0x7f6d480057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 -- 172.21.15.46:0/1205531859 shutdown_connections 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 --2- 172.21.15.46:0/1205531859 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d480047c0 0x7f6d48004c40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 -- 172.21.15.46:0/1205531859 >> 172.21.15.46:0/1205531859 conn(0x7f6d6006a4d0 msgr2=0x7f6d6006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 -- 172.21.15.46:0/1205531859 shutdown_connections 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.887+0000 7f6d66b61640 1 -- 172.21.15.46:0/1205531859 wait complete. 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.888+0000 7f6d66b61640 1 Processor -- start 2023-12-16T21:14:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.888+0000 7f6d66b61640 1 -- start start 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d66b61640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d480047c0 0x7f6d600ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d66b61640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 0x7f6d60143850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d66b61640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d600ab970 0x7f6d6015b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d66b61640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600ac910 con 0x7f6d600ad9b0 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d66b61640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600acbf0 con 0x7f6d480047c0 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d66b61640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d600aced0 con 0x7f6d600ab970 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 0x7f6d60143850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 0x7f6d60143850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41712/0 (socket says 172.21.15.46:41712) 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 -- 172.21.15.46:0/3010756055 learned_addr learned my addr 172.21.15.46:0/3010756055 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d650fe640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d600ab970 0x7f6d6015b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d648fd640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d480047c0 0x7f6d600ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 -- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d600ab970 msgr2=0x7f6d6015b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d600ab970 0x7f6d6015b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 -- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d480047c0 msgr2=0x7f6d600ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d480047c0 0x7f6d600ac280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.889+0000 7f6d5ffff640 1 -- 172.21.15.46:0/3010756055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d600ad1b0 con 0x7f6d600ad9b0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.890+0000 7f6d5ffff640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 0x7f6d60143850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d5402c380 tx=0x7f6d54001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.890+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5403c040 con 0x7f6d600ad9b0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.890+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5402cbf0 con 0x7f6d600ad9b0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.890+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d54007070 con 0x7f6d600ad9b0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.890+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d600ad490 con 0x7f6d600ad9b0 2023-12-16T21:14:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.890+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6015bcc0 con 0x7f6d600ad9b0 2023-12-16T21:14:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.891+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d600ad620 con 0x7f6d600ad9b0 2023-12-16T21:14:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.891+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6d54032070 con 0x7f6d600ad9b0 2023-12-16T21:14:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.892+0000 7f6d5d7fa640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6d4c06ae70 0x7f6d4c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.892+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d540aa9b0 con 0x7f6d600ad9b0 2023-12-16T21:14:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.892+0000 7f6d648fd640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6d4c06ae70 0x7f6d4c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.893+0000 7f6d648fd640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6d4c06ae70 0x7f6d4c06d330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6d58000f80 tx=0x7f6d58009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:38.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:38.896+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6d54075a60 con 0x7f6d600ad9b0 2023-12-16T21:14:39.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.108+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d60069400 con 0x7f6d600ad9b0 2023-12-16T21:14:39.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.109+0000 7f6d5d7fa640 1 -- 172.21.15.46:0/3010756055 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f6d54079080 con 0x7f6d600ad9b0 2023-12-16T21:14:39.109 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:39.109 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:38.102533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:14:39.110 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2023-12-16T21:14:39.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6d4c06ae70 msgr2=0x7f6d4c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6d4c06ae70 0x7f6d4c06d330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6d58000f80 tx=0x7f6d58009040 comp rx=0 tx=0).stop 2023-12-16T21:14:39.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 msgr2=0x7f6d60143850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 0x7f6d60143850 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6d5402c380 tx=0x7f6d54001980 comp rx=0 tx=0).stop 2023-12-16T21:14:39.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 shutdown_connections 2023-12-16T21:14:39.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6d4c06ae70 0x7f6d4c06d330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d600ad9b0 0x7f6d60143850 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d600ab970 0x7f6d6015b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 --2- 172.21.15.46:0/3010756055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d480047c0 0x7f6d600ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.110+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 >> 172.21.15.46:0/3010756055 conn(0x7f6d6006a4d0 msgr2=0x7f6d6006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.111+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 shutdown_connections 2023-12-16T21:14:39.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.111+0000 7f6d66b61640 1 -- 172.21.15.46:0/3010756055 wait complete. 2023-12-16T21:14:39.122 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.228+0000 7f048a58e640 1 Processor -- start 2023-12-16T21:14:39.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.228+0000 7f048a58e640 1 -- start start 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.228+0000 7f048a58e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04840d2910 0x7f04840d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.228+0000 7f048a58e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840b2b10 con 0x7f048415d330 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.228+0000 7f048a58e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840d1590 con 0x7f04840d3730 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.228+0000 7f048a58e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840d3bc0 con 0x7f04840d2910 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0483fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f048415d330 0x7f04840b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36606/0 (socket says 172.21.15.46:36606) 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0483fff640 1 -- 172.21.15.46:0/3774674967 learned_addr learned my addr 172.21.15.46:0/3774674967 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f04837fe640 1 --2- 172.21.15.46:0/3774674967 >> v2:172.21.15.121:3300/0 conn(0x7f04840d2910 0x7f04840d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402421989 0 0) 0x7f04840b2b10 con 0x7f048415d330 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046c003520 con 0x7f048415d330 2023-12-16T21:14:39.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1950356217 0 0) 0x7f046c003520 con 0x7f048415d330 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 >> v2:172.21.15.121:3300/0 conn(0x7f04840d2910 msgr2=0x7f04840d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3774674967 >> v2:172.21.15.121:3300/0 conn(0x7f04840d2910 0x7f04840d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 >> v1:172.21.15.121:6789/0 conn(0x7f04840d3730 legacy=0x7f04840b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840d1870 con 0x7f048415d330 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f0474001670 con 0x7f048415d330 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 >> v1:172.21.15.46:6789/0 conn(0x7f048415d330 legacy=0x7f04840b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c003970 0x7f046c003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c004370 0x7f046c004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c0050e0 0x7f046c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04840d3bc0 con 0x7f046c004370 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c00c020 con 0x7f046c003970 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c00c1e0 con 0x7f046c0050e0 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f04837fe640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c004370 0x7f046c004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0483fff640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c003970 0x7f046c003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.229+0000 7f0488b2b640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c0050e0 0x7f046c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f04837fe640 1 -- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c0050e0 msgr2=0x7f046c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f04837fe640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c0050e0 0x7f046c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f04837fe640 1 -- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c003970 msgr2=0x7f046c003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f04837fe640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c003970 0x7f046c003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f04837fe640 1 -- 172.21.15.46:0/3774674967 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c003520 con 0x7f046c004370 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f0488b2b640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c0050e0 0x7f046c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f0488b2b640 1 -- 172.21.15.46:0/3774674967 reap_dead start 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.230+0000 7f04837fe640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c004370 0x7f046c004840 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0478009ac0 tx=0x7f046c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e006c50bb8cd8fc1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047802c6b0 con 0x7f046c004370 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047802c810 con 0x7f046c004370 2023-12-16T21:14:39.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f0482ffd640 1 -- 172.21.15.46:0/3774674967 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047802cba0 con 0x7f046c004370 2023-12-16T21:14:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 -- 172.21.15.46:0/3774674967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c004370 msgr2=0x7f046c004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c004370 0x7f046c004840 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0478009ac0 tx=0x7f046c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 -- 172.21.15.46:0/3774674967 shutdown_connections 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 --2- 172.21.15.46:0/3774674967 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c004370 0x7f046c004840 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 -- 172.21.15.46:0/3774674967 >> 172.21.15.46:0/3774674967 conn(0x7f0484072020 msgr2=0x7f0484072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 -- 172.21.15.46:0/3774674967 shutdown_connections 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.231+0000 7f048a58e640 1 -- 172.21.15.46:0/3774674967 wait complete. 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.232+0000 7f048a58e640 1 Processor -- start 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 -- start start 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c004370 0x7f04841513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048415ac10 0x7f048415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 0x7f0484152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048415b5d0 con 0x7f0484151bb0 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0484152570 con 0x7f046c004370 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f048a58e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048415da60 con 0x7f048415ac10 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f0488b2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 0x7f0484152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.237 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f0488b2b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 0x7f0484152030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41734/0 (socket says 172.21.15.46:41734) 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.233+0000 7f0488b2b640 1 -- 172.21.15.46:0/1190344926 learned_addr learned my addr 172.21.15.46:0/1190344926 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0483fff640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c004370 0x7f04841513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f04837fe640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048415ac10 0x7f048415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0488b2b640 1 -- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048415ac10 msgr2=0x7f048415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0488b2b640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048415ac10 0x7f048415b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0488b2b640 1 -- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c004370 msgr2=0x7f04841513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0488b2b640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c004370 0x7f04841513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0488b2b640 1 -- 172.21.15.46:0/1190344926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04841f2d80 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0488b2b640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 0x7f0484152030 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f04840cebd0 tx=0x7f047c00ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c006b50 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c006cb0 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0484152790 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c003bb0 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.234+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048415ed90 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.235+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0484069400 con 0x7f0484151bb0 2023-12-16T21:14:39.238 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.235+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f047c003d10 con 0x7f0484151bb0 2023-12-16T21:14:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.236+0000 7f0480ff9640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f047006ae70 0x7f047006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.236+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f047c088b40 con 0x7f0484151bb0 2023-12-16T21:14:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.236+0000 7f0483fff640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f047006ae70 0x7f047006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.239 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.236+0000 7f0483fff640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f047006ae70 0x7f047006d330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0474007c20 tx=0x7f0474009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.240 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.240+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f047c057660 con 0x7f0484151bb0 2023-12-16T21:14:39.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.418+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04840c9280 con 0x7f0484151bb0 2023-12-16T21:14:39.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.419+0000 7f0480ff9640 1 -- 172.21.15.46:0/1190344926 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f047c0571f0 con 0x7f0484151bb0 2023-12-16T21:14:39.420 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:39.420 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":34,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:37.491321+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f047006ae70 msgr2=0x7f047006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f047006ae70 0x7f047006d330 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0474007c20 tx=0x7f0474009040 comp rx=0 tx=0).stop 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 msgr2=0x7f0484152030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 0x7f0484152030 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f04840cebd0 tx=0x7f047c00ebe0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 shutdown_connections 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f047006ae70 0x7f047006d330 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0484151bb0 0x7f0484152030 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048415ac10 0x7f048415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 --2- 172.21.15.46:0/1190344926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c004370 0x7f04841513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 >> 172.21.15.46:0/1190344926 conn(0x7f0484072020 msgr2=0x7f04840b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 shutdown_connections 2023-12-16T21:14:39.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.421+0000 7f048a58e640 1 -- 172.21.15.46:0/1190344926 wait complete. 2023-12-16T21:14:39.433 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-16T21:14:39.433 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:14:39.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.539+0000 7f28b8dc9640 1 Processor -- start 2023-12-16T21:14:39.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.539+0000 7f28b8dc9640 1 -- start start 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b8dc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b40b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b8dc9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f28b406f1c0 0x7f28b406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b8dc9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b4146fd0 con 0x7f28b4146840 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b8dc9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b406c180 con 0x7f28b406f1c0 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b8dc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b406c460 con 0x7f28b4146460 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b40b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b2d9d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f28b406f1c0 0x7f28b406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b40b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:47304/0 (socket says 172.21.15.46:47304) 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 -- 172.21.15.46:0/3483158210 learned_addr learned my addr 172.21.15.46:0/3483158210 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28abfff640 1 --1- 172.21.15.46:0/3483158210 >> v1:172.21.15.121:6789/0 conn(0x7f28b4146840 0x7f28b406eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:35354/0 (socket says 172.21.15.46:35354) 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 -- 172.21.15.46:0/3483158210 >> v2:172.21.15.121:3300/0 conn(0x7f28b406f1c0 msgr2=0x7f28b406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 --2- 172.21.15.46:0/3483158210 >> v2:172.21.15.121:3300/0 conn(0x7f28b406f1c0 0x7f28b406baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 -- 172.21.15.46:0/3483158210 >> v1:172.21.15.121:6789/0 conn(0x7f28b4146840 legacy=0x7f28b406eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.540+0000 7f28b259c640 1 -- 172.21.15.46:0/3483158210 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b406c740 con 0x7f28b4146460 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.541+0000 7f28b259c640 1 --2- 172.21.15.46:0/3483158210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b40b49d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009870 tx=0x7f28a402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=776bab399dbfaddc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.541+0000 7f28b1d9b640 1 -- 172.21.15.46:0/3483158210 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4030840 con 0x7f28b4146460 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.541+0000 7f28b1d9b640 1 -- 172.21.15.46:0/3483158210 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a40309a0 con 0x7f28b4146460 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.541+0000 7f28b1d9b640 1 -- 172.21.15.46:0/3483158210 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4030d30 con 0x7f28b4146460 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 -- 172.21.15.46:0/3483158210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 msgr2=0x7f28b40b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/3483158210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b40b49d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009870 tx=0x7f28a402b980 comp rx=0 tx=0).stop 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 -- 172.21.15.46:0/3483158210 shutdown_connections 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/3483158210 >> v2:172.21.15.121:3300/0 conn(0x7f28b406f1c0 0x7f28b406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/3483158210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b40b49d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 -- 172.21.15.46:0/3483158210 >> 172.21.15.46:0/3483158210 conn(0x7f28b406a6b0 msgr2=0x7f28b406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 -- 172.21.15.46:0/3483158210 shutdown_connections 2023-12-16T21:14:39.543 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.542+0000 7f28b8dc9640 1 -- 172.21.15.46:0/3483158210 wait complete. 2023-12-16T21:14:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.543+0000 7f28b8dc9640 1 Processor -- start 2023-12-16T21:14:39.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 -- start start 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 0x7f28b4140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b414e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28b4146840 0x7f28b414c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b4140c90 con 0x7f28b406f1c0 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b414e830 con 0x7f28b4146840 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b8dc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b414de60 con 0x7f28b4146460 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 0x7f28b4140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b259c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 0x7f28b4140750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41750/0 (socket says 172.21.15.46:41750) 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.544+0000 7f28b259c640 1 -- 172.21.15.46:0/2189805864 learned_addr learned my addr 172.21.15.46:0/2189805864 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28abfff640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b414e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b2d9d640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28b4146840 0x7f28b414c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b259c640 1 -- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 msgr2=0x7f28b414e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b259c640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b414e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b259c640 1 -- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28b4146840 msgr2=0x7f28b414c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b259c640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28b4146840 0x7f28b414c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b259c640 1 -- 172.21.15.46:0/2189805864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a40094d0 con 0x7f28b406f1c0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b259c640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 0x7f28b4140750 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f28a402b9c0 tx=0x7f28a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4039c90 con 0x7f28b406f1c0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a4039df0 con 0x7f28b406f1c0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b414d250 con 0x7f28b406f1c0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a40446c0 con 0x7f28b406f1c0 2023-12-16T21:14:39.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.545+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b414d810 con 0x7f28b406f1c0 2023-12-16T21:14:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.546+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b4069400 con 0x7f28b406f1c0 2023-12-16T21:14:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.546+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f28a404d3b0 con 0x7f28b406f1c0 2023-12-16T21:14:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.547+0000 7f28aaffd640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f289406ae40 0x7f289406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.547+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28a403e070 con 0x7f28b406f1c0 2023-12-16T21:14:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.547+0000 7f28abfff640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f289406ae40 0x7f289406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.548+0000 7f28abfff640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f289406ae40 0x7f289406d300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f28b406d250 tx=0x7f289c000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.551 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.551+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f28a40771f0 con 0x7f28b406f1c0 2023-12-16T21:14:39.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.722+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f28b406cd00 con 0x7f289406ae40 2023-12-16T21:14:39.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.730+0000 7f28aaffd640 1 -- 172.21.15.46:0/2189805864 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274324 (secure 0 0 0) 0x7f289c004c70 con 0x7f289406ae40 2023-12-16T21:14:39.734 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:39.736 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f289406ae40 msgr2=0x7f289406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f289406ae40 0x7f289406d300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f28b406d250 tx=0x7f289c000ed0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 msgr2=0x7f28b4140750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 0x7f28b4140750 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f28a402b9c0 tx=0x7f28a4004730 comp rx=0 tx=0).stop 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 shutdown_connections 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f289406ae40 0x7f289406d300 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28b4146840 0x7f28b414c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28b4146460 0x7f28b414e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 --2- 172.21.15.46:0/2189805864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f28b406f1c0 0x7f28b4140750 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 >> 172.21.15.46:0/2189805864 conn(0x7f28b406a6b0 msgr2=0x7f28b40b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 shutdown_connections 2023-12-16T21:14:39.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.733+0000 7f28b8dc9640 1 -- 172.21.15.46:0/2189805864 wait complete. 2023-12-16T21:14:39.745 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2023-12-16T21:14:37.969382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":236660,"kb_used_data":3784,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738060,"statfs":{"total":773094113280,"available":772851773440,"internally_reserved":0,"allocated":3874816,"data_stored":2189504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000802"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:35.729092+0000","last_change":"2023-12-16T21:14:35.729092+0000","last_active":"2023-12-16T21:14:35.729092+0000","last_peered":"2023-12-16T21:14:35.729092+0000","last_clean":"2023-12-16T21:14:35.729092+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:35.729092+0000","last_undegraded":"2023-12-16T21:14:35.729092+0000","last_fullsized":"2023-12-16T21:14:35.729092+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-16T21:14:34.723664+0000","last_change":"2023-12-16T21:14:34.723664+0000","last_active":"2023-12-16T21:14:34.723664+0000","last_peered":"2023-12-16T21:14:34.723664+0000","last_clean":"2023-12-16T21:14:34.723664+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-16T21:14:34.723664+0000","last_undegraded":"2023-12-16T21:14:34.723664+0000","last_fullsized":"2023-12-16T21:14:34.723664+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":172,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-12-16T21:14:30.891636+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:14:30.891636+0000","last_peered":"2023-12-16T21:14:30.891636+0000","last_clean":"2023-12-16T21:14:30.891636+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:14:30.891636+0000","last_undegraded":"2023-12-16T21:14:30.891636+0000","last_fullsized":"2023-12-16T21:14:30.891636+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":929792,"data_stored":716464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.50700000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.433}]},{"osd":3,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.625}]},{"osd":4,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.48299999999999998}]},{"osd":5,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.69699999999999995}]},{"osd":6,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.61099999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27356,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":335872,"data_stored":126096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60299999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.73599999999999999}]},{"osd":2,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.628},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.69399999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.75},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.70699999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.64300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":335872,"data_stored":126096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":5.3639999999999999,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67100000000000004}]},{"osd":4,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.57799999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.48399999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.498}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27352,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":335872,"data_stored":126096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52300000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.504}]},{"osd":5,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.47199999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57199999999999995}]},{"osd":7,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61399999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31520,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":335872,"data_stored":126096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.375}]},{"osd":1,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.249},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.313}]},{"osd":2,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.35299999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":1.046},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":1.091}]},{"osd":5,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.96999999999999997}]},{"osd":6,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":1.008},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.0569999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.93700000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31516,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":335872,"data_stored":126096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.34599999999999997}]},{"osd":1,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.38400000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.36299999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.66600000000000004}]},{"osd":5,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.746},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.61199999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68000000000000005}]},{"osd":7,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63}]}]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31524,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":335872,"data_stored":126096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.309},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.35399999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.33300000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.54300000000000004}]},{"osd":5,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.52500000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.59599999999999997}]},{"osd":7,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.627}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":929792,"data_stored":716464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.42699999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.69899999999999995}]},{"osd":6,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.629}]},{"osd":7,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.71499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:14:39.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:14:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.891+0000 7fb53cce7640 1 Processor -- start 2023-12-16T21:14:39.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.891+0000 7fb53cce7640 1 -- start start 2023-12-16T21:14:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb53cce7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb53815d330 0x7fb5380b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb53cce7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5380d3900 0x7fb5380b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb53cce7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380b2b10 con 0x7fb5380d2910 2023-12-16T21:14:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb53cce7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380d1590 con 0x7fb5380d3900 2023-12-16T21:14:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb53cce7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380d1870 con 0x7fb53815d330 2023-12-16T21:14:39.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb536d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5380d3900 0x7fb5380b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb535d9b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb5380d2910 0x7fb5380d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36616/0 (socket says 172.21.15.46:36616) 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb535d9b640 1 -- 172.21.15.46:0/2190265216 learned_addr learned my addr 172.21.15.46:0/2190265216 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.892+0000 7fb536d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb5380d3900 0x7fb5380b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41764/0 (socket says 172.21.15.46:41764) 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb53659c640 1 --2- 172.21.15.46:0/2190265216 >> v2:172.21.15.121:3300/0 conn(0x7fb53815d330 0x7fb5380b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb536d9d640 1 -- 172.21.15.46:0/2190265216 >> v2:172.21.15.121:3300/0 conn(0x7fb53815d330 msgr2=0x7fb5380b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb536d9d640 1 --2- 172.21.15.46:0/2190265216 >> v2:172.21.15.121:3300/0 conn(0x7fb53815d330 0x7fb5380b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb536d9d640 1 -- 172.21.15.46:0/2190265216 >> v1:172.21.15.46:6789/0 conn(0x7fb5380d2910 legacy=0x7fb5380d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb53559a640 1 -- 172.21.15.46:0/2190265216 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364018312 0 0) 0x7fb5380b2b10 con 0x7fb5380d2910 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb536d9d640 1 -- 172.21.15.46:0/2190265216 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5380d1b50 con 0x7fb5380d3900 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.893+0000 7fb53659c640 1 --2- 172.21.15.46:0/2190265216 >> v2:172.21.15.121:3300/0 conn(0x7fb53815d330 0x7fb5380b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb536d9d640 1 --2- 172.21.15.46:0/2190265216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d3900 0x7fb5380b6390 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb5380d0d10 tx=0x7fb52802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86d266e6a3d64d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53559a640 1 -- 172.21.15.46:0/2190265216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb52802cbf0 con 0x7fb5380d3900 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53559a640 1 -- 172.21.15.46:0/2190265216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb52802cd50 con 0x7fb5380d3900 2023-12-16T21:14:39.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53559a640 1 -- 172.21.15.46:0/2190265216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb528034660 con 0x7fb5380d3900 2023-12-16T21:14:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 -- 172.21.15.46:0/2190265216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d3900 msgr2=0x7fb5380b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.896 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 --2- 172.21.15.46:0/2190265216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d3900 0x7fb5380b6390 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb5380d0d10 tx=0x7fb52802be40 comp rx=0 tx=0).stop 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 -- 172.21.15.46:0/2190265216 shutdown_connections 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 --2- 172.21.15.46:0/2190265216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d3900 0x7fb5380b6390 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 --2- 172.21.15.46:0/2190265216 >> v2:172.21.15.121:3300/0 conn(0x7fb53815d330 0x7fb5380b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 -- 172.21.15.46:0/2190265216 >> 172.21.15.46:0/2190265216 conn(0x7fb538072020 msgr2=0x7fb538072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 -- 172.21.15.46:0/2190265216 shutdown_connections 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.894+0000 7fb53cce7640 1 -- 172.21.15.46:0/2190265216 wait complete. 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 Processor -- start 2023-12-16T21:14:39.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 -- start start 2023-12-16T21:14:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 0x7fb53814b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5380d3900 0x7fb5380c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.899 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb53815d330 0x7fb53815f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815ef40 con 0x7fb5380d2910 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.896+0000 7fb53cce7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53814bb50 con 0x7fb53815d330 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53cce7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53815e6a0 con 0x7fb5380d3900 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 0x7fb53814b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 0x7fb53814b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41768/0 (socket says 172.21.15.46:41768) 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 -- 172.21.15.46:0/3542874583 learned_addr learned my addr 172.21.15.46:0/3542874583 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb535d9b640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5380d3900 0x7fb5380c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb536d9d640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb53815d330 0x7fb53815f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 -- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5380d3900 msgr2=0x7fb5380c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5380d3900 0x7fb5380c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 -- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb53815d330 msgr2=0x7fb53815f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb53815d330 0x7fb53815f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 -- 172.21.15.46:0/3542874583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528009a70 con 0x7fb5380d2910 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb536d9d640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb53815d330 0x7fb53815f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:39.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53659c640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 0x7fb53814b610 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e7e0 tx=0x7fb53000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530004030 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb530004190 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb530004520 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53815e980 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.897+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380c6cd0 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.898+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538069400 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.899+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb53000c6d0 con 0x7fb5380d2910 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.899+0000 7fb526ffd640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb51806ae70 0x7fb51806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:39.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.899+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb530089220 con 0x7fb5380d2910 2023-12-16T21:14:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.899+0000 7fb535d9b640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb51806ae70 0x7fb51806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:39.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.900+0000 7fb535d9b640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb51806ae70 0x7fb51806d330 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5380b1e60 tx=0x7fb52c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:39.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:39.903+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb530054360 con 0x7fb5380d2910 2023-12-16T21:14:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.075+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5380b4910 con 0x7fb51806ae70 2023-12-16T21:14:40.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.082+0000 7fb526ffd640 1 -- 172.21.15.46:0/3542874583 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289091 (secure 0 0 0) 0x7fb5380b4910 con 0x7fb51806ae70 2023-12-16T21:14:40.086 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:40.087 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb51806ae70 msgr2=0x7fb51806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb51806ae70 0x7fb51806d330 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb5380b1e60 tx=0x7fb52c009040 comp rx=0 tx=0).stop 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 msgr2=0x7fb53814b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 0x7fb53814b610 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb53000e7e0 tx=0x7fb53000eca0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 shutdown_connections 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb51806ae70 0x7fb51806d330 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb53815d330 0x7fb53815f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5380d3900 0x7fb5380c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 --2- 172.21.15.46:0/3542874583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5380d2910 0x7fb53814b610 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 >> 172.21.15.46:0/3542874583 conn(0x7fb538072020 msgr2=0x7fb5380d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 shutdown_connections 2023-12-16T21:14:40.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.085+0000 7fb53cce7640 1 -- 172.21.15.46:0/3542874583 wait complete. 2023-12-16T21:14:40.097 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2023-12-16T21:14:39.969787+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236976,"kb_used_data":4148,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737744,"statfs":{"total":773094113280,"available":772851449856,"internally_reserved":0,"allocated":4247552,"data_stored":2351496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.239916"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.507738+0000","last_change":"2023-12-16T21:14:36.505214+0000","last_active":"2023-12-16T21:14:36.507738+0000","last_peered":"2023-12-16T21:14:36.507738+0000","last_clean":"2023-12-16T21:14:36.507738+0000","last_became_active":"2023-12-16T21:14:36.504943+0000","last_became_peered":"2023-12-16T21:14:36.504943+0000","last_unstale":"2023-12-16T21:14:36.507738+0000","last_undegraded":"2023-12-16T21:14:36.507738+0000","last_fullsized":"2023-12-16T21:14:36.507738+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:10:40.156523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494592+0000","last_change":"2023-12-16T21:14:36.508945+0000","last_active":"2023-12-16T21:14:37.494592+0000","last_peered":"2023-12-16T21:14:37.494592+0000","last_clean":"2023-12-16T21:14:37.494592+0000","last_became_active":"2023-12-16T21:14:36.508663+0000","last_became_peered":"2023-12-16T21:14:36.508663+0000","last_unstale":"2023-12-16T21:14:37.494592+0000","last_undegraded":"2023-12-16T21:14:37.494592+0000","last_fullsized":"2023-12-16T21:14:37.494592+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:23:06.701480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498357+0000","last_change":"2023-12-16T21:14:36.507694+0000","last_active":"2023-12-16T21:14:37.498357+0000","last_peered":"2023-12-16T21:14:37.498357+0000","last_clean":"2023-12-16T21:14:37.498357+0000","last_became_active":"2023-12-16T21:14:36.506857+0000","last_became_peered":"2023-12-16T21:14:36.506857+0000","last_unstale":"2023-12-16T21:14:37.498357+0000","last_undegraded":"2023-12-16T21:14:37.498357+0000","last_fullsized":"2023-12-16T21:14:37.498357+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:51:43.686250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.516906+0000","last_change":"2023-12-16T21:14:36.516202+0000","last_active":"2023-12-16T21:14:36.516906+0000","last_peered":"2023-12-16T21:14:36.516906+0000","last_clean":"2023-12-16T21:14:36.516906+0000","last_became_active":"2023-12-16T21:14:36.515982+0000","last_became_peered":"2023-12-16T21:14:36.515982+0000","last_unstale":"2023-12-16T21:14:36.516906+0000","last_undegraded":"2023-12-16T21:14:36.516906+0000","last_fullsized":"2023-12-16T21:14:36.516906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:40:25.598672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.507164+0000","last_change":"2023-12-16T21:14:36.505823+0000","last_active":"2023-12-16T21:14:36.507164+0000","last_peered":"2023-12-16T21:14:36.507164+0000","last_clean":"2023-12-16T21:14:36.507164+0000","last_became_active":"2023-12-16T21:14:36.505524+0000","last_became_peered":"2023-12-16T21:14:36.505524+0000","last_unstale":"2023-12-16T21:14:36.507164+0000","last_undegraded":"2023-12-16T21:14:36.507164+0000","last_fullsized":"2023-12-16T21:14:36.507164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:37:39.072716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494462+0000","last_change":"2023-12-16T21:14:36.506034+0000","last_active":"2023-12-16T21:14:37.494462+0000","last_peered":"2023-12-16T21:14:37.494462+0000","last_clean":"2023-12-16T21:14:37.494462+0000","last_became_active":"2023-12-16T21:14:36.505202+0000","last_became_peered":"2023-12-16T21:14:36.505202+0000","last_unstale":"2023-12-16T21:14:37.494462+0000","last_undegraded":"2023-12-16T21:14:37.494462+0000","last_fullsized":"2023-12-16T21:14:37.494462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:03:02.309527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.499006+0000","last_change":"2023-12-16T21:14:36.509308+0000","last_active":"2023-12-16T21:14:37.499006+0000","last_peered":"2023-12-16T21:14:37.499006+0000","last_clean":"2023-12-16T21:14:37.499006+0000","last_became_active":"2023-12-16T21:14:36.509015+0000","last_became_peered":"2023-12-16T21:14:36.509015+0000","last_unstale":"2023-12-16T21:14:37.499006+0000","last_undegraded":"2023-12-16T21:14:37.499006+0000","last_fullsized":"2023-12-16T21:14:37.499006+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:23:06.701480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498143+0000","last_change":"2023-12-16T21:14:36.504716+0000","last_active":"2023-12-16T21:14:37.498143+0000","last_peered":"2023-12-16T21:14:37.498143+0000","last_clean":"2023-12-16T21:14:37.498143+0000","last_became_active":"2023-12-16T21:14:36.504486+0000","last_became_peered":"2023-12-16T21:14:36.504486+0000","last_unstale":"2023-12-16T21:14:37.498143+0000","last_undegraded":"2023-12-16T21:14:37.498143+0000","last_fullsized":"2023-12-16T21:14:37.498143+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:27:31.991848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.517277+0000","last_change":"2023-12-16T21:14:36.511502+0000","last_active":"2023-12-16T21:14:36.517277+0000","last_peered":"2023-12-16T21:14:36.517277+0000","last_clean":"2023-12-16T21:14:36.517277+0000","last_became_active":"2023-12-16T21:14:36.511216+0000","last_became_peered":"2023-12-16T21:14:36.511216+0000","last_unstale":"2023-12-16T21:14:36.517277+0000","last_undegraded":"2023-12-16T21:14:36.517277+0000","last_fullsized":"2023-12-16T21:14:36.517277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:07:06.226100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.511234+0000","last_change":"2023-12-16T21:14:36.510133+0000","last_active":"2023-12-16T21:14:36.511234+0000","last_peered":"2023-12-16T21:14:36.511234+0000","last_clean":"2023-12-16T21:14:36.511234+0000","last_became_active":"2023-12-16T21:14:36.509872+0000","last_became_peered":"2023-12-16T21:14:36.509872+0000","last_unstale":"2023-12-16T21:14:36.511234+0000","last_undegraded":"2023-12-16T21:14:36.511234+0000","last_fullsized":"2023-12-16T21:14:36.511234+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:47:07.892691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.499006+0000","last_change":"2023-12-16T21:14:36.512209+0000","last_active":"2023-12-16T21:14:37.499006+0000","last_peered":"2023-12-16T21:14:37.499006+0000","last_clean":"2023-12-16T21:14:37.499006+0000","last_became_active":"2023-12-16T21:14:36.511909+0000","last_became_peered":"2023-12-16T21:14:36.511909+0000","last_unstale":"2023-12-16T21:14:37.499006+0000","last_undegraded":"2023-12-16T21:14:37.499006+0000","last_fullsized":"2023-12-16T21:14:37.499006+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:03:02.309527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498602+0000","last_change":"2023-12-16T21:14:36.504267+0000","last_active":"2023-12-16T21:14:37.498602+0000","last_peered":"2023-12-16T21:14:37.498602+0000","last_clean":"2023-12-16T21:14:37.498602+0000","last_became_active":"2023-12-16T21:14:36.503974+0000","last_became_peered":"2023-12-16T21:14:36.503974+0000","last_unstale":"2023-12-16T21:14:37.498602+0000","last_undegraded":"2023-12-16T21:14:37.498602+0000","last_fullsized":"2023-12-16T21:14:37.498602+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:48:59.695070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.508959+0000","last_change":"2023-12-16T21:14:36.508306+0000","last_active":"2023-12-16T21:14:36.508959+0000","last_peered":"2023-12-16T21:14:36.508959+0000","last_clean":"2023-12-16T21:14:36.508959+0000","last_became_active":"2023-12-16T21:14:36.508029+0000","last_became_peered":"2023-12-16T21:14:36.508029+0000","last_unstale":"2023-12-16T21:14:36.508959+0000","last_undegraded":"2023-12-16T21:14:36.508959+0000","last_fullsized":"2023-12-16T21:14:36.508959+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:07:29.082199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494601+0000","last_change":"2023-12-16T21:14:36.502746+0000","last_active":"2023-12-16T21:14:37.494601+0000","last_peered":"2023-12-16T21:14:37.494601+0000","last_clean":"2023-12-16T21:14:37.494601+0000","last_became_active":"2023-12-16T21:14:36.502412+0000","last_became_peered":"2023-12-16T21:14:36.502412+0000","last_unstale":"2023-12-16T21:14:37.494601+0000","last_undegraded":"2023-12-16T21:14:37.494601+0000","last_fullsized":"2023-12-16T21:14:37.494601+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:58:24.162996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.499136+0000","last_change":"2023-12-16T21:14:36.506919+0000","last_active":"2023-12-16T21:14:37.499136+0000","last_peered":"2023-12-16T21:14:37.499136+0000","last_clean":"2023-12-16T21:14:37.499136+0000","last_became_active":"2023-12-16T21:14:36.506611+0000","last_became_peered":"2023-12-16T21:14:36.506611+0000","last_unstale":"2023-12-16T21:14:37.499136+0000","last_undegraded":"2023-12-16T21:14:37.499136+0000","last_fullsized":"2023-12-16T21:14:37.499136+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:36:03.278033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498603+0000","last_change":"2023-12-16T21:14:36.503507+0000","last_active":"2023-12-16T21:14:37.498603+0000","last_peered":"2023-12-16T21:14:37.498603+0000","last_clean":"2023-12-16T21:14:37.498603+0000","last_became_active":"2023-12-16T21:14:36.503195+0000","last_became_peered":"2023-12-16T21:14:36.503195+0000","last_unstale":"2023-12-16T21:14:37.498603+0000","last_undegraded":"2023-12-16T21:14:37.498603+0000","last_fullsized":"2023-12-16T21:14:37.498603+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:48:59.695070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498874+0000","last_change":"2023-12-16T21:14:36.509521+0000","last_active":"2023-12-16T21:14:37.498874+0000","last_peered":"2023-12-16T21:14:37.498874+0000","last_clean":"2023-12-16T21:14:37.498874+0000","last_became_active":"2023-12-16T21:14:36.508640+0000","last_became_peered":"2023-12-16T21:14:36.508640+0000","last_unstale":"2023-12-16T21:14:37.498874+0000","last_undegraded":"2023-12-16T21:14:37.498874+0000","last_fullsized":"2023-12-16T21:14:37.498874+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:00:03.357856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494692+0000","last_change":"2023-12-16T21:14:36.510985+0000","last_active":"2023-12-16T21:14:37.494692+0000","last_peered":"2023-12-16T21:14:37.494692+0000","last_clean":"2023-12-16T21:14:37.494692+0000","last_became_active":"2023-12-16T21:14:36.510707+0000","last_became_peered":"2023-12-16T21:14:36.510707+0000","last_unstale":"2023-12-16T21:14:37.494692+0000","last_undegraded":"2023-12-16T21:14:37.494692+0000","last_fullsized":"2023-12-16T21:14:37.494692+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:15:37.198468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494460+0000","last_change":"2023-12-16T21:14:36.510962+0000","last_active":"2023-12-16T21:14:37.494460+0000","last_peered":"2023-12-16T21:14:37.494460+0000","last_clean":"2023-12-16T21:14:37.494460+0000","last_became_active":"2023-12-16T21:14:36.510690+0000","last_became_peered":"2023-12-16T21:14:36.510690+0000","last_unstale":"2023-12-16T21:14:37.494460+0000","last_undegraded":"2023-12-16T21:14:37.494460+0000","last_fullsized":"2023-12-16T21:14:37.494460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:48:39.795292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.504482+0000","last_change":"2023-12-16T21:14:36.503366+0000","last_active":"2023-12-16T21:14:36.504482+0000","last_peered":"2023-12-16T21:14:36.504482+0000","last_clean":"2023-12-16T21:14:36.504482+0000","last_became_active":"2023-12-16T21:14:36.503120+0000","last_became_peered":"2023-12-16T21:14:36.503120+0000","last_unstale":"2023-12-16T21:14:36.504482+0000","last_undegraded":"2023-12-16T21:14:36.504482+0000","last_fullsized":"2023-12-16T21:14:36.504482+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:11:31.838134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.517214+0000","last_change":"2023-12-16T21:14:36.512897+0000","last_active":"2023-12-16T21:14:36.517214+0000","last_peered":"2023-12-16T21:14:36.517214+0000","last_clean":"2023-12-16T21:14:36.517214+0000","last_became_active":"2023-12-16T21:14:36.512656+0000","last_became_peered":"2023-12-16T21:14:36.512656+0000","last_unstale":"2023-12-16T21:14:36.517214+0000","last_undegraded":"2023-12-16T21:14:36.517214+0000","last_fullsized":"2023-12-16T21:14:36.517214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:47:36.697618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.517146+0000","last_change":"2023-12-16T21:14:36.511971+0000","last_active":"2023-12-16T21:14:36.517146+0000","last_peered":"2023-12-16T21:14:36.517146+0000","last_clean":"2023-12-16T21:14:36.517146+0000","last_became_active":"2023-12-16T21:14:36.511643+0000","last_became_peered":"2023-12-16T21:14:36.511643+0000","last_unstale":"2023-12-16T21:14:36.517146+0000","last_undegraded":"2023-12-16T21:14:36.517146+0000","last_fullsized":"2023-12-16T21:14:36.517146+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:20.096815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.506075+0000","last_change":"2023-12-16T21:14:36.504343+0000","last_active":"2023-12-16T21:14:36.506075+0000","last_peered":"2023-12-16T21:14:36.506075+0000","last_clean":"2023-12-16T21:14:36.506075+0000","last_became_active":"2023-12-16T21:14:36.504070+0000","last_became_peered":"2023-12-16T21:14:36.504070+0000","last_unstale":"2023-12-16T21:14:36.506075+0000","last_undegraded":"2023-12-16T21:14:36.506075+0000","last_fullsized":"2023-12-16T21:14:36.506075+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:14:43.407787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.499071+0000","last_change":"2023-12-16T21:14:36.499603+0000","last_active":"2023-12-16T21:14:37.499071+0000","last_peered":"2023-12-16T21:14:37.499071+0000","last_clean":"2023-12-16T21:14:37.499071+0000","last_became_active":"2023-12-16T21:14:36.499327+0000","last_became_peered":"2023-12-16T21:14:36.499327+0000","last_unstale":"2023-12-16T21:14:37.499071+0000","last_undegraded":"2023-12-16T21:14:37.499071+0000","last_fullsized":"2023-12-16T21:14:37.499071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:58:24.162996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.509987+0000","last_change":"2023-12-16T21:14:36.508546+0000","last_active":"2023-12-16T21:14:36.509987+0000","last_peered":"2023-12-16T21:14:36.509987+0000","last_clean":"2023-12-16T21:14:36.509987+0000","last_became_active":"2023-12-16T21:14:36.508277+0000","last_became_peered":"2023-12-16T21:14:36.508277+0000","last_unstale":"2023-12-16T21:14:36.509987+0000","last_undegraded":"2023-12-16T21:14:36.509987+0000","last_fullsized":"2023-12-16T21:14:36.509987+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:09:27.338191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494734+0000","last_change":"2023-12-16T21:14:36.507928+0000","last_active":"2023-12-16T21:14:37.494734+0000","last_peered":"2023-12-16T21:14:37.494734+0000","last_clean":"2023-12-16T21:14:37.494734+0000","last_became_active":"2023-12-16T21:14:36.507675+0000","last_became_peered":"2023-12-16T21:14:36.507675+0000","last_unstale":"2023-12-16T21:14:37.494734+0000","last_undegraded":"2023-12-16T21:14:37.494734+0000","last_fullsized":"2023-12-16T21:14:37.494734+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:13:08.336399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.514664+0000","last_change":"2023-12-16T21:14:36.513458+0000","last_active":"2023-12-16T21:14:36.514664+0000","last_peered":"2023-12-16T21:14:36.514664+0000","last_clean":"2023-12-16T21:14:36.514664+0000","last_became_active":"2023-12-16T21:14:36.513152+0000","last_became_peered":"2023-12-16T21:14:36.513152+0000","last_unstale":"2023-12-16T21:14:36.514664+0000","last_undegraded":"2023-12-16T21:14:36.514664+0000","last_fullsized":"2023-12-16T21:14:36.514664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:19:41.010178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498736+0000","last_change":"2023-12-16T21:14:36.500898+0000","last_active":"2023-12-16T21:14:37.498736+0000","last_peered":"2023-12-16T21:14:37.498736+0000","last_clean":"2023-12-16T21:14:37.498736+0000","last_became_active":"2023-12-16T21:14:36.500640+0000","last_became_peered":"2023-12-16T21:14:36.500640+0000","last_unstale":"2023-12-16T21:14:37.498736+0000","last_undegraded":"2023-12-16T21:14:37.498736+0000","last_fullsized":"2023-12-16T21:14:37.498736+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:48:39.795292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.519674+0000","last_change":"2023-12-16T21:14:36.516153+0000","last_active":"2023-12-16T21:14:36.519674+0000","last_peered":"2023-12-16T21:14:36.519674+0000","last_clean":"2023-12-16T21:14:36.519674+0000","last_became_active":"2023-12-16T21:14:36.515667+0000","last_became_peered":"2023-12-16T21:14:36.515667+0000","last_unstale":"2023-12-16T21:14:36.519674+0000","last_undegraded":"2023-12-16T21:14:36.519674+0000","last_fullsized":"2023-12-16T21:14:36.519674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:55.281295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498427+0000","last_change":"2023-12-16T21:14:36.508898+0000","last_active":"2023-12-16T21:14:37.498427+0000","last_peered":"2023-12-16T21:14:37.498427+0000","last_clean":"2023-12-16T21:14:37.498427+0000","last_became_active":"2023-12-16T21:14:36.508590+0000","last_became_peered":"2023-12-16T21:14:36.508590+0000","last_unstale":"2023-12-16T21:14:37.498427+0000","last_undegraded":"2023-12-16T21:14:37.498427+0000","last_fullsized":"2023-12-16T21:14:37.498427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:20:25.344935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.510972+0000","last_change":"2023-12-16T21:14:36.506361+0000","last_active":"2023-12-16T21:14:36.510972+0000","last_peered":"2023-12-16T21:14:36.510972+0000","last_clean":"2023-12-16T21:14:36.510972+0000","last_became_active":"2023-12-16T21:14:36.506112+0000","last_became_peered":"2023-12-16T21:14:36.506112+0000","last_unstale":"2023-12-16T21:14:36.510972+0000","last_undegraded":"2023-12-16T21:14:36.510972+0000","last_fullsized":"2023-12-16T21:14:36.510972+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:51:40.787437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498219+0000","last_change":"2023-12-16T21:14:36.515460+0000","last_active":"2023-12-16T21:14:37.498219+0000","last_peered":"2023-12-16T21:14:37.498219+0000","last_clean":"2023-12-16T21:14:37.498219+0000","last_became_active":"2023-12-16T21:14:36.514970+0000","last_became_peered":"2023-12-16T21:14:36.514970+0000","last_unstale":"2023-12-16T21:14:37.498219+0000","last_undegraded":"2023-12-16T21:14:37.498219+0000","last_fullsized":"2023-12-16T21:14:37.498219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:15:02.363501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.506116+0000","last_change":"2023-12-16T21:14:36.503688+0000","last_active":"2023-12-16T21:14:36.506116+0000","last_peered":"2023-12-16T21:14:36.506116+0000","last_clean":"2023-12-16T21:14:36.506116+0000","last_became_active":"2023-12-16T21:14:36.503376+0000","last_became_peered":"2023-12-16T21:14:36.503376+0000","last_unstale":"2023-12-16T21:14:36.506116+0000","last_undegraded":"2023-12-16T21:14:36.506116+0000","last_fullsized":"2023-12-16T21:14:36.506116+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:18:34.632138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758123+0000","last_change":"2023-12-16T21:14:35.749881+0000","last_active":"2023-12-16T21:14:36.758123+0000","last_peered":"2023-12-16T21:14:36.758123+0000","last_clean":"2023-12-16T21:14:36.758123+0000","last_became_active":"2023-12-16T21:14:35.749618+0000","last_became_peered":"2023-12-16T21:14:35.749618+0000","last_unstale":"2023-12-16T21:14:36.758123+0000","last_undegraded":"2023-12-16T21:14:36.758123+0000","last_fullsized":"2023-12-16T21:14:36.758123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:19.083707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.519368+0000","last_change":"2023-12-16T21:14:36.516262+0000","last_active":"2023-12-16T21:14:36.519368+0000","last_peered":"2023-12-16T21:14:36.519368+0000","last_clean":"2023-12-16T21:14:36.519368+0000","last_became_active":"2023-12-16T21:14:36.515986+0000","last_became_peered":"2023-12-16T21:14:36.515986+0000","last_unstale":"2023-12-16T21:14:36.519368+0000","last_undegraded":"2023-12-16T21:14:36.519368+0000","last_fullsized":"2023-12-16T21:14:36.519368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:14:43.407787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758682+0000","last_change":"2023-12-16T21:14:35.749342+0000","last_active":"2023-12-16T21:14:36.758682+0000","last_peered":"2023-12-16T21:14:36.758682+0000","last_clean":"2023-12-16T21:14:36.758682+0000","last_became_active":"2023-12-16T21:14:35.748813+0000","last_became_peered":"2023-12-16T21:14:35.748813+0000","last_unstale":"2023-12-16T21:14:36.758682+0000","last_undegraded":"2023-12-16T21:14:36.758682+0000","last_fullsized":"2023-12-16T21:14:36.758682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:19:40.004750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.519562+0000","last_change":"2023-12-16T21:14:36.515108+0000","last_active":"2023-12-16T21:14:36.519562+0000","last_peered":"2023-12-16T21:14:36.519562+0000","last_clean":"2023-12-16T21:14:36.519562+0000","last_became_active":"2023-12-16T21:14:36.514846+0000","last_became_peered":"2023-12-16T21:14:36.514846+0000","last_unstale":"2023-12-16T21:14:36.519562+0000","last_undegraded":"2023-12-16T21:14:36.519562+0000","last_fullsized":"2023-12-16T21:14:36.519562+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:14:43.407787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494712+0000","last_change":"2023-12-16T21:14:35.752384+0000","last_active":"2023-12-16T21:14:37.494712+0000","last_peered":"2023-12-16T21:14:37.494712+0000","last_clean":"2023-12-16T21:14:37.494712+0000","last_became_active":"2023-12-16T21:14:35.752111+0000","last_became_peered":"2023-12-16T21:14:35.752111+0000","last_unstale":"2023-12-16T21:14:37.494712+0000","last_undegraded":"2023-12-16T21:14:37.494712+0000","last_fullsized":"2023-12-16T21:14:37.494712+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:41:09.561781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.514644+0000","last_change":"2023-12-16T21:14:36.513420+0000","last_active":"2023-12-16T21:14:36.514644+0000","last_peered":"2023-12-16T21:14:36.514644+0000","last_clean":"2023-12-16T21:14:36.514644+0000","last_became_active":"2023-12-16T21:14:36.513127+0000","last_became_peered":"2023-12-16T21:14:36.513127+0000","last_unstale":"2023-12-16T21:14:36.514644+0000","last_undegraded":"2023-12-16T21:14:36.514644+0000","last_fullsized":"2023-12-16T21:14:36.514644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:20.089135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.757107+0000","last_change":"2023-12-16T21:14:35.751044+0000","last_active":"2023-12-16T21:14:36.757107+0000","last_peered":"2023-12-16T21:14:36.757107+0000","last_clean":"2023-12-16T21:14:36.757107+0000","last_became_active":"2023-12-16T21:14:35.750712+0000","last_became_peered":"2023-12-16T21:14:35.750712+0000","last_unstale":"2023-12-16T21:14:36.757107+0000","last_undegraded":"2023-12-16T21:14:36.757107+0000","last_fullsized":"2023-12-16T21:14:36.757107+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:19.091387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498550+0000","last_change":"2023-12-16T21:14:36.515498+0000","last_active":"2023-12-16T21:14:37.498550+0000","last_peered":"2023-12-16T21:14:37.498550+0000","last_clean":"2023-12-16T21:14:37.498550+0000","last_became_active":"2023-12-16T21:14:36.515122+0000","last_became_peered":"2023-12-16T21:14:36.515122+0000","last_unstale":"2023-12-16T21:14:37.498550+0000","last_undegraded":"2023-12-16T21:14:37.498550+0000","last_fullsized":"2023-12-16T21:14:37.498550+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:51:43.686250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758371+0000","last_change":"2023-12-16T21:14:35.752730+0000","last_active":"2023-12-16T21:14:36.758371+0000","last_peered":"2023-12-16T21:14:36.758371+0000","last_clean":"2023-12-16T21:14:36.758371+0000","last_became_active":"2023-12-16T21:14:35.752444+0000","last_became_peered":"2023-12-16T21:14:35.752444+0000","last_unstale":"2023-12-16T21:14:36.758371+0000","last_undegraded":"2023-12-16T21:14:36.758371+0000","last_fullsized":"2023-12-16T21:14:36.758371+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:19:40.004750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498346+0000","last_change":"2023-12-16T21:14:36.499682+0000","last_active":"2023-12-16T21:14:37.498346+0000","last_peered":"2023-12-16T21:14:37.498346+0000","last_clean":"2023-12-16T21:14:37.498346+0000","last_became_active":"2023-12-16T21:14:36.499399+0000","last_became_peered":"2023-12-16T21:14:36.499399+0000","last_unstale":"2023-12-16T21:14:37.498346+0000","last_undegraded":"2023-12-16T21:14:37.498346+0000","last_fullsized":"2023-12-16T21:14:37.498346+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:20:25.344935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758350+0000","last_change":"2023-12-16T21:14:35.752740+0000","last_active":"2023-12-16T21:14:36.758350+0000","last_peered":"2023-12-16T21:14:36.758350+0000","last_clean":"2023-12-16T21:14:36.758350+0000","last_became_active":"2023-12-16T21:14:35.752454+0000","last_became_peered":"2023-12-16T21:14:35.752454+0000","last_unstale":"2023-12-16T21:14:36.758350+0000","last_undegraded":"2023-12-16T21:14:36.758350+0000","last_fullsized":"2023-12-16T21:14:36.758350+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:03:17.623707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.517120+0000","last_change":"2023-12-16T21:14:36.512415+0000","last_active":"2023-12-16T21:14:36.517120+0000","last_peered":"2023-12-16T21:14:36.517120+0000","last_clean":"2023-12-16T21:14:36.517120+0000","last_became_active":"2023-12-16T21:14:36.511296+0000","last_became_peered":"2023-12-16T21:14:36.511296+0000","last_unstale":"2023-12-16T21:14:36.517120+0000","last_undegraded":"2023-12-16T21:14:36.517120+0000","last_fullsized":"2023-12-16T21:14:36.517120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:37:39.072716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494296+0000","last_change":"2023-12-16T21:14:35.744534+0000","last_active":"2023-12-16T21:14:37.494296+0000","last_peered":"2023-12-16T21:14:37.494296+0000","last_clean":"2023-12-16T21:14:37.494296+0000","last_became_active":"2023-12-16T21:14:35.744252+0000","last_became_peered":"2023-12-16T21:14:35.744252+0000","last_unstale":"2023-12-16T21:14:37.494296+0000","last_undegraded":"2023-12-16T21:14:37.494296+0000","last_fullsized":"2023-12-16T21:14:37.494296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:48:58.689643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.509592+0000","last_change":"2023-12-16T21:14:36.506005+0000","last_active":"2023-12-16T21:14:36.509592+0000","last_peered":"2023-12-16T21:14:36.509592+0000","last_clean":"2023-12-16T21:14:36.509592+0000","last_became_active":"2023-12-16T21:14:36.505742+0000","last_became_peered":"2023-12-16T21:14:36.505742+0000","last_unstale":"2023-12-16T21:14:36.509592+0000","last_undegraded":"2023-12-16T21:14:36.509592+0000","last_fullsized":"2023-12-16T21:14:36.509592+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:18:34.632138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.508827+0000","last_change":"2023-12-16T21:14:35.746954+0000","last_active":"2023-12-16T21:14:36.508827+0000","last_peered":"2023-12-16T21:14:36.508827+0000","last_clean":"2023-12-16T21:14:36.508827+0000","last_became_active":"2023-12-16T21:14:35.746505+0000","last_became_peered":"2023-12-16T21:14:35.746505+0000","last_unstale":"2023-12-16T21:14:36.508827+0000","last_undegraded":"2023-12-16T21:14:36.508827+0000","last_fullsized":"2023-12-16T21:14:36.508827+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:18:33.626710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498605+0000","last_change":"2023-12-16T21:14:36.505580+0000","last_active":"2023-12-16T21:14:37.498605+0000","last_peered":"2023-12-16T21:14:37.498605+0000","last_clean":"2023-12-16T21:14:37.498605+0000","last_became_active":"2023-12-16T21:14:36.505109+0000","last_became_peered":"2023-12-16T21:14:36.505109+0000","last_unstale":"2023-12-16T21:14:37.498605+0000","last_undegraded":"2023-12-16T21:14:37.498605+0000","last_fullsized":"2023-12-16T21:14:37.498605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:00:03.357856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494623+0000","last_change":"2023-12-16T21:14:35.749726+0000","last_active":"2023-12-16T21:14:37.494623+0000","last_peered":"2023-12-16T21:14:37.494623+0000","last_clean":"2023-12-16T21:14:37.494623+0000","last_became_active":"2023-12-16T21:14:35.748329+0000","last_became_peered":"2023-12-16T21:14:35.748329+0000","last_unstale":"2023-12-16T21:14:37.494623+0000","last_undegraded":"2023-12-16T21:14:37.494623+0000","last_fullsized":"2023-12-16T21:14:37.494623+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:15:01.358073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494716+0000","last_change":"2023-12-16T21:14:36.510248+0000","last_active":"2023-12-16T21:14:37.494716+0000","last_peered":"2023-12-16T21:14:37.494716+0000","last_clean":"2023-12-16T21:14:37.494716+0000","last_became_active":"2023-12-16T21:14:36.510003+0000","last_became_peered":"2023-12-16T21:14:36.510003+0000","last_unstale":"2023-12-16T21:14:37.494716+0000","last_undegraded":"2023-12-16T21:14:37.494716+0000","last_fullsized":"2023-12-16T21:14:37.494716+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:01:43.715199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.493522+0000","last_change":"2023-12-16T21:14:35.749129+0000","last_active":"2023-12-16T21:14:36.493522+0000","last_peered":"2023-12-16T21:14:36.493522+0000","last_clean":"2023-12-16T21:14:36.493522+0000","last_became_active":"2023-12-16T21:14:35.748319+0000","last_became_peered":"2023-12-16T21:14:35.748319+0000","last_unstale":"2023-12-16T21:14:36.493522+0000","last_undegraded":"2023-12-16T21:14:36.493522+0000","last_fullsized":"2023-12-16T21:14:36.493522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:41:09.561781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494751+0000","last_change":"2023-12-16T21:14:36.512606+0000","last_active":"2023-12-16T21:14:37.494751+0000","last_peered":"2023-12-16T21:14:37.494751+0000","last_clean":"2023-12-16T21:14:37.494751+0000","last_became_active":"2023-12-16T21:14:36.511597+0000","last_became_peered":"2023-12-16T21:14:36.511597+0000","last_unstale":"2023-12-16T21:14:37.494751+0000","last_undegraded":"2023-12-16T21:14:37.494751+0000","last_fullsized":"2023-12-16T21:14:37.494751+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:15:44.877163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498446+0000","last_change":"2023-12-16T21:14:35.749095+0000","last_active":"2023-12-16T21:14:37.498446+0000","last_peered":"2023-12-16T21:14:37.498446+0000","last_clean":"2023-12-16T21:14:37.498446+0000","last_became_active":"2023-12-16T21:14:35.748683+0000","last_became_peered":"2023-12-16T21:14:35.748683+0000","last_unstale":"2023-12-16T21:14:37.498446+0000","last_undegraded":"2023-12-16T21:14:37.498446+0000","last_fullsized":"2023-12-16T21:14:37.498446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:27:30.986421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498219+0000","last_change":"2023-12-16T21:14:36.507759+0000","last_active":"2023-12-16T21:14:37.498219+0000","last_peered":"2023-12-16T21:14:37.498219+0000","last_clean":"2023-12-16T21:14:37.498219+0000","last_became_active":"2023-12-16T21:14:36.507504+0000","last_became_peered":"2023-12-16T21:14:36.507504+0000","last_unstale":"2023-12-16T21:14:37.498219+0000","last_undegraded":"2023-12-16T21:14:37.498219+0000","last_fullsized":"2023-12-16T21:14:37.498219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:33:09.479821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494862+0000","last_change":"2023-12-16T21:14:35.753302+0000","last_active":"2023-12-16T21:14:37.494862+0000","last_peered":"2023-12-16T21:14:37.494862+0000","last_clean":"2023-12-16T21:14:37.494862+0000","last_became_active":"2023-12-16T21:14:35.751958+0000","last_became_peered":"2023-12-16T21:14:35.751958+0000","last_unstale":"2023-12-16T21:14:37.494862+0000","last_undegraded":"2023-12-16T21:14:37.494862+0000","last_fullsized":"2023-12-16T21:14:37.494862+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:33:08.474393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.511197+0000","last_change":"2023-12-16T21:14:36.509888+0000","last_active":"2023-12-16T21:14:36.511197+0000","last_peered":"2023-12-16T21:14:36.511197+0000","last_clean":"2023-12-16T21:14:36.511197+0000","last_became_active":"2023-12-16T21:14:36.509646+0000","last_became_peered":"2023-12-16T21:14:36.509646+0000","last_unstale":"2023-12-16T21:14:36.511197+0000","last_undegraded":"2023-12-16T21:14:36.511197+0000","last_fullsized":"2023-12-16T21:14:36.511197+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:41:10.567208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758758+0000","last_change":"2023-12-16T21:14:35.746417+0000","last_active":"2023-12-16T21:14:36.758758+0000","last_peered":"2023-12-16T21:14:36.758758+0000","last_clean":"2023-12-16T21:14:36.758758+0000","last_became_active":"2023-12-16T21:14:35.746126+0000","last_became_peered":"2023-12-16T21:14:35.746126+0000","last_unstale":"2023-12-16T21:14:36.758758+0000","last_undegraded":"2023-12-16T21:14:36.758758+0000","last_fullsized":"2023-12-16T21:14:36.758758+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:41:09.561781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494666+0000","last_change":"2023-12-16T21:14:36.509044+0000","last_active":"2023-12-16T21:14:37.494666+0000","last_peered":"2023-12-16T21:14:37.494666+0000","last_clean":"2023-12-16T21:14:37.494666+0000","last_became_active":"2023-12-16T21:14:36.508689+0000","last_became_peered":"2023-12-16T21:14:36.508689+0000","last_unstale":"2023-12-16T21:14:37.494666+0000","last_undegraded":"2023-12-16T21:14:37.494666+0000","last_fullsized":"2023-12-16T21:14:37.494666+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:36:03.278033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.501614+0000","last_change":"2023-12-16T21:14:35.750157+0000","last_active":"2023-12-16T21:14:36.501614+0000","last_peered":"2023-12-16T21:14:36.501614+0000","last_clean":"2023-12-16T21:14:36.501614+0000","last_became_active":"2023-12-16T21:14:35.749872+0000","last_became_peered":"2023-12-16T21:14:35.749872+0000","last_unstale":"2023-12-16T21:14:36.501614+0000","last_undegraded":"2023-12-16T21:14:36.501614+0000","last_fullsized":"2023-12-16T21:14:36.501614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:41.715235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498352+0000","last_change":"2023-12-16T21:14:36.513311+0000","last_active":"2023-12-16T21:14:37.498352+0000","last_peered":"2023-12-16T21:14:37.498352+0000","last_clean":"2023-12-16T21:14:37.498352+0000","last_became_active":"2023-12-16T21:14:36.512924+0000","last_became_peered":"2023-12-16T21:14:36.512924+0000","last_unstale":"2023-12-16T21:14:37.498352+0000","last_undegraded":"2023-12-16T21:14:37.498352+0000","last_fullsized":"2023-12-16T21:14:37.498352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:33:09.479821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494239+0000","last_change":"2023-12-16T21:14:35.751589+0000","last_active":"2023-12-16T21:14:37.494239+0000","last_peered":"2023-12-16T21:14:37.494239+0000","last_clean":"2023-12-16T21:14:37.494239+0000","last_became_active":"2023-12-16T21:14:35.751308+0000","last_became_peered":"2023-12-16T21:14:35.751308+0000","last_unstale":"2023-12-16T21:14:37.494239+0000","last_undegraded":"2023-12-16T21:14:37.494239+0000","last_fullsized":"2023-12-16T21:14:37.494239+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:51:42.680822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.516825+0000","last_change":"2023-12-16T21:14:36.513141+0000","last_active":"2023-12-16T21:14:36.516825+0000","last_peered":"2023-12-16T21:14:36.516825+0000","last_clean":"2023-12-16T21:14:36.516825+0000","last_became_active":"2023-12-16T21:14:36.512868+0000","last_became_peered":"2023-12-16T21:14:36.512868+0000","last_unstale":"2023-12-16T21:14:36.516825+0000","last_undegraded":"2023-12-16T21:14:36.516825+0000","last_fullsized":"2023-12-16T21:14:36.516825+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:17:55.281295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494272+0000","last_change":"2023-12-16T21:14:35.753089+0000","last_active":"2023-12-16T21:14:37.494272+0000","last_peered":"2023-12-16T21:14:37.494272+0000","last_clean":"2023-12-16T21:14:37.494272+0000","last_became_active":"2023-12-16T21:14:35.751900+0000","last_became_peered":"2023-12-16T21:14:35.751900+0000","last_unstale":"2023-12-16T21:14:37.494272+0000","last_undegraded":"2023-12-16T21:14:37.494272+0000","last_fullsized":"2023-12-16T21:14:37.494272+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:28:40.167067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.519533+0000","last_change":"2023-12-16T21:14:36.514189+0000","last_active":"2023-12-16T21:14:36.519533+0000","last_peered":"2023-12-16T21:14:36.519533+0000","last_clean":"2023-12-16T21:14:36.519533+0000","last_became_active":"2023-12-16T21:14:36.513533+0000","last_became_peered":"2023-12-16T21:14:36.513533+0000","last_unstale":"2023-12-16T21:14:36.519533+0000","last_undegraded":"2023-12-16T21:14:36.519533+0000","last_fullsized":"2023-12-16T21:14:36.519533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:51:40.787437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.756979+0000","last_change":"2023-12-16T21:14:35.755201+0000","last_active":"2023-12-16T21:14:36.756979+0000","last_peered":"2023-12-16T21:14:36.756979+0000","last_clean":"2023-12-16T21:14:36.756979+0000","last_became_active":"2023-12-16T21:14:35.754937+0000","last_became_peered":"2023-12-16T21:14:35.754937+0000","last_unstale":"2023-12-16T21:14:36.756979+0000","last_undegraded":"2023-12-16T21:14:36.756979+0000","last_fullsized":"2023-12-16T21:14:36.756979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:28:40.167067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498826+0000","last_change":"2023-12-16T21:14:36.515923+0000","last_active":"2023-12-16T21:14:37.498826+0000","last_peered":"2023-12-16T21:14:37.498826+0000","last_clean":"2023-12-16T21:14:37.498826+0000","last_became_active":"2023-12-16T21:14:36.515022+0000","last_became_peered":"2023-12-16T21:14:36.515022+0000","last_unstale":"2023-12-16T21:14:37.498826+0000","last_undegraded":"2023-12-16T21:14:37.498826+0000","last_fullsized":"2023-12-16T21:14:37.498826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:03:02.309527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758532+0000","last_change":"2023-12-16T21:14:35.746690+0000","last_active":"2023-12-16T21:14:36.758532+0000","last_peered":"2023-12-16T21:14:36.758532+0000","last_clean":"2023-12-16T21:14:36.758532+0000","last_became_active":"2023-12-16T21:14:35.746278+0000","last_became_peered":"2023-12-16T21:14:35.746278+0000","last_unstale":"2023-12-16T21:14:36.758532+0000","last_undegraded":"2023-12-16T21:14:36.758532+0000","last_fullsized":"2023-12-16T21:14:36.758532+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:19.083707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.516917+0000","last_change":"2023-12-16T21:14:36.516278+0000","last_active":"2023-12-16T21:14:36.516917+0000","last_peered":"2023-12-16T21:14:36.516917+0000","last_clean":"2023-12-16T21:14:36.516917+0000","last_became_active":"2023-12-16T21:14:36.515937+0000","last_became_peered":"2023-12-16T21:14:36.515937+0000","last_unstale":"2023-12-16T21:14:36.516917+0000","last_undegraded":"2023-12-16T21:14:36.516917+0000","last_fullsized":"2023-12-16T21:14:36.516917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:21:09.689650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.758384+0000","last_change":"2023-12-16T21:14:35.745150+0000","last_active":"2023-12-16T21:14:36.758384+0000","last_peered":"2023-12-16T21:14:36.758384+0000","last_clean":"2023-12-16T21:14:36.758384+0000","last_became_active":"2023-12-16T21:14:35.744867+0000","last_became_peered":"2023-12-16T21:14:35.744867+0000","last_unstale":"2023-12-16T21:14:36.758384+0000","last_undegraded":"2023-12-16T21:14:36.758384+0000","last_fullsized":"2023-12-16T21:14:36.758384+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:09:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.504543+0000","last_change":"2023-12-16T21:14:36.503725+0000","last_active":"2023-12-16T21:14:36.504543+0000","last_peered":"2023-12-16T21:14:36.504543+0000","last_clean":"2023-12-16T21:14:36.504543+0000","last_became_active":"2023-12-16T21:14:36.503437+0000","last_became_peered":"2023-12-16T21:14:36.503437+0000","last_unstale":"2023-12-16T21:14:36.504543+0000","last_undegraded":"2023-12-16T21:14:36.504543+0000","last_fullsized":"2023-12-16T21:14:36.504543+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:54:13.741992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.508203+0000","last_change":"2023-12-16T21:14:35.748223+0000","last_active":"2023-12-16T21:14:36.508203+0000","last_peered":"2023-12-16T21:14:36.508203+0000","last_clean":"2023-12-16T21:14:36.508203+0000","last_became_active":"2023-12-16T21:14:35.747959+0000","last_became_peered":"2023-12-16T21:14:35.747959+0000","last_unstale":"2023-12-16T21:14:36.508203+0000","last_undegraded":"2023-12-16T21:14:36.508203+0000","last_fullsized":"2023-12-16T21:14:36.508203+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:10:39.151095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.515911+0000","last_change":"2023-12-16T21:14:36.515390+0000","last_active":"2023-12-16T21:14:36.515911+0000","last_peered":"2023-12-16T21:14:36.515911+0000","last_clean":"2023-12-16T21:14:36.515911+0000","last_became_active":"2023-12-16T21:14:36.515110+0000","last_became_peered":"2023-12-16T21:14:36.515110+0000","last_unstale":"2023-12-16T21:14:36.515911+0000","last_undegraded":"2023-12-16T21:14:36.515911+0000","last_fullsized":"2023-12-16T21:14:36.515911+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:51:17.051861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.757357+0000","last_change":"2023-12-16T21:14:35.750048+0000","last_active":"2023-12-16T21:14:36.757357+0000","last_peered":"2023-12-16T21:14:36.757357+0000","last_clean":"2023-12-16T21:14:36.757357+0000","last_became_active":"2023-12-16T21:14:35.749788+0000","last_became_peered":"2023-12-16T21:14:35.749788+0000","last_unstale":"2023-12-16T21:14:36.757357+0000","last_undegraded":"2023-12-16T21:14:36.757357+0000","last_fullsized":"2023-12-16T21:14:36.757357+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:09:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.503205+0000","last_change":"2023-12-16T21:14:36.501379+0000","last_active":"2023-12-16T21:14:36.503205+0000","last_peered":"2023-12-16T21:14:36.503205+0000","last_clean":"2023-12-16T21:14:36.503205+0000","last_became_active":"2023-12-16T21:14:36.501119+0000","last_became_peered":"2023-12-16T21:14:36.501119+0000","last_unstale":"2023-12-16T21:14:36.503205+0000","last_undegraded":"2023-12-16T21:14:36.503205+0000","last_fullsized":"2023-12-16T21:14:36.503205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:30:25.944342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.499914+0000","last_change":"2023-12-16T21:14:35.751891+0000","last_active":"2023-12-16T21:14:36.499914+0000","last_peered":"2023-12-16T21:14:36.499914+0000","last_clean":"2023-12-16T21:14:36.499914+0000","last_became_active":"2023-12-16T21:14:35.751543+0000","last_became_peered":"2023-12-16T21:14:35.751543+0000","last_unstale":"2023-12-16T21:14:36.499914+0000","last_undegraded":"2023-12-16T21:14:36.499914+0000","last_fullsized":"2023-12-16T21:14:36.499914+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:09:26.332763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.501118+0000","last_change":"2023-12-16T21:14:36.497823+0000","last_active":"2023-12-16T21:14:36.501118+0000","last_peered":"2023-12-16T21:14:36.501118+0000","last_clean":"2023-12-16T21:14:36.501118+0000","last_became_active":"2023-12-16T21:14:36.497534+0000","last_became_peered":"2023-12-16T21:14:36.497534+0000","last_unstale":"2023-12-16T21:14:36.501118+0000","last_undegraded":"2023-12-16T21:14:36.501118+0000","last_fullsized":"2023-12-16T21:14:36.501118+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:10:15.419267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.509751+0000","last_change":"2023-12-16T21:14:35.749753+0000","last_active":"2023-12-16T21:14:36.509751+0000","last_peered":"2023-12-16T21:14:36.509751+0000","last_clean":"2023-12-16T21:14:36.509751+0000","last_became_active":"2023-12-16T21:14:35.748444+0000","last_became_peered":"2023-12-16T21:14:35.748444+0000","last_unstale":"2023-12-16T21:14:36.509751+0000","last_undegraded":"2023-12-16T21:14:36.509751+0000","last_fullsized":"2023-12-16T21:14:36.509751+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:41.715235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498929+0000","last_change":"2023-12-16T21:14:36.507383+0000","last_active":"2023-12-16T21:14:37.498929+0000","last_peered":"2023-12-16T21:14:37.498929+0000","last_clean":"2023-12-16T21:14:37.498929+0000","last_became_active":"2023-12-16T21:14:36.507125+0000","last_became_peered":"2023-12-16T21:14:36.507125+0000","last_unstale":"2023-12-16T21:14:37.498929+0000","last_undegraded":"2023-12-16T21:14:37.498929+0000","last_fullsized":"2023-12-16T21:14:37.498929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:48:39.795292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498064+0000","last_change":"2023-12-16T21:14:35.748936+0000","last_active":"2023-12-16T21:14:37.498064+0000","last_peered":"2023-12-16T21:14:37.498064+0000","last_clean":"2023-12-16T21:14:37.498064+0000","last_became_active":"2023-12-16T21:14:35.748510+0000","last_became_peered":"2023-12-16T21:14:35.748510+0000","last_unstale":"2023-12-16T21:14:37.498064+0000","last_undegraded":"2023-12-16T21:14:37.498064+0000","last_fullsized":"2023-12-16T21:14:37.498064+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:21:08.684223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.504199+0000","last_change":"2023-12-16T21:14:36.501956+0000","last_active":"2023-12-16T21:14:36.504199+0000","last_peered":"2023-12-16T21:14:36.504199+0000","last_clean":"2023-12-16T21:14:36.504199+0000","last_became_active":"2023-12-16T21:14:36.501662+0000","last_became_peered":"2023-12-16T21:14:36.501662+0000","last_unstale":"2023-12-16T21:14:36.504199+0000","last_undegraded":"2023-12-16T21:14:36.504199+0000","last_fullsized":"2023-12-16T21:14:36.504199+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:28:38.343549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.755996+0000","last_change":"2023-12-16T21:14:35.752093+0000","last_active":"2023-12-16T21:14:36.755996+0000","last_peered":"2023-12-16T21:14:36.755996+0000","last_clean":"2023-12-16T21:14:36.755996+0000","last_became_active":"2023-12-16T21:14:35.751817+0000","last_became_peered":"2023-12-16T21:14:35.751817+0000","last_unstale":"2023-12-16T21:14:36.755996+0000","last_undegraded":"2023-12-16T21:14:36.755996+0000","last_fullsized":"2023-12-16T21:14:36.755996+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:57:38.342980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.498953+0000","last_change":"2023-12-16T21:14:36.512060+0000","last_active":"2023-12-16T21:14:37.498953+0000","last_peered":"2023-12-16T21:14:37.498953+0000","last_clean":"2023-12-16T21:14:37.498953+0000","last_became_active":"2023-12-16T21:14:36.511759+0000","last_became_peered":"2023-12-16T21:14:36.511759+0000","last_unstale":"2023-12-16T21:14:37.498953+0000","last_undegraded":"2023-12-16T21:14:37.498953+0000","last_fullsized":"2023-12-16T21:14:37.498953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:23:06.701480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.493642+0000","last_change":"2023-12-16T21:14:35.749541+0000","last_active":"2023-12-16T21:14:36.493642+0000","last_peered":"2023-12-16T21:14:36.493642+0000","last_clean":"2023-12-16T21:14:36.493642+0000","last_became_active":"2023-12-16T21:14:35.748782+0000","last_became_peered":"2023-12-16T21:14:35.748782+0000","last_unstale":"2023-12-16T21:14:36.493642+0000","last_undegraded":"2023-12-16T21:14:36.493642+0000","last_fullsized":"2023-12-16T21:14:36.493642+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:51:41.715235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.513022+0000","last_change":"2023-12-16T21:14:36.512048+0000","last_active":"2023-12-16T21:14:36.513022+0000","last_peered":"2023-12-16T21:14:36.513022+0000","last_clean":"2023-12-16T21:14:36.513022+0000","last_became_active":"2023-12-16T21:14:36.511754+0000","last_became_peered":"2023-12-16T21:14:36.511754+0000","last_unstale":"2023-12-16T21:14:36.513022+0000","last_undegraded":"2023-12-16T21:14:36.513022+0000","last_fullsized":"2023-12-16T21:14:36.513022+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:03:18.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.504841+0000","last_change":"2023-12-16T21:14:35.752228+0000","last_active":"2023-12-16T21:14:36.504841+0000","last_peered":"2023-12-16T21:14:36.504841+0000","last_clean":"2023-12-16T21:14:36.504841+0000","last_became_active":"2023-12-16T21:14:35.751934+0000","last_became_peered":"2023-12-16T21:14:35.751934+0000","last_unstale":"2023-12-16T21:14:36.504841+0000","last_undegraded":"2023-12-16T21:14:36.504841+0000","last_fullsized":"2023-12-16T21:14:36.504841+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:28.784238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.506071+0000","last_change":"2023-12-16T21:14:36.502515+0000","last_active":"2023-12-16T21:14:36.506071+0000","last_peered":"2023-12-16T21:14:36.506071+0000","last_clean":"2023-12-16T21:14:36.506071+0000","last_became_active":"2023-12-16T21:14:36.502237+0000","last_became_peered":"2023-12-16T21:14:36.502237+0000","last_unstale":"2023-12-16T21:14:36.506071+0000","last_undegraded":"2023-12-16T21:14:36.506071+0000","last_fullsized":"2023-12-16T21:14:36.506071+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:03:18.629135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.493481+0000","last_change":"2023-12-16T21:14:35.747300+0000","last_active":"2023-12-16T21:14:36.493481+0000","last_peered":"2023-12-16T21:14:36.493481+0000","last_clean":"2023-12-16T21:14:36.493481+0000","last_became_active":"2023-12-16T21:14:35.746884+0000","last_became_peered":"2023-12-16T21:14:35.746884+0000","last_unstale":"2023-12-16T21:14:36.493481+0000","last_undegraded":"2023-12-16T21:14:36.493481+0000","last_fullsized":"2023-12-16T21:14:36.493481+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:53:19.083707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.504573+0000","last_change":"2023-12-16T21:14:36.500322+0000","last_active":"2023-12-16T21:14:36.504573+0000","last_peered":"2023-12-16T21:14:36.504573+0000","last_clean":"2023-12-16T21:14:36.504573+0000","last_became_active":"2023-12-16T21:14:36.500053+0000","last_became_peered":"2023-12-16T21:14:36.500053+0000","last_unstale":"2023-12-16T21:14:36.504573+0000","last_undegraded":"2023-12-16T21:14:36.504573+0000","last_fullsized":"2023-12-16T21:14:36.504573+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:10:40.156523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.501761+0000","last_change":"2023-12-16T21:14:35.755176+0000","last_active":"2023-12-16T21:14:36.501761+0000","last_peered":"2023-12-16T21:14:36.501761+0000","last_clean":"2023-12-16T21:14:36.501761+0000","last_became_active":"2023-12-16T21:14:35.754914+0000","last_became_peered":"2023-12-16T21:14:35.754914+0000","last_unstale":"2023-12-16T21:14:36.501761+0000","last_undegraded":"2023-12-16T21:14:36.501761+0000","last_fullsized":"2023-12-16T21:14:36.501761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:56:09.194348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494247+0000","last_change":"2023-12-16T21:14:36.511443+0000","last_active":"2023-12-16T21:14:37.494247+0000","last_peered":"2023-12-16T21:14:37.494247+0000","last_clean":"2023-12-16T21:14:37.494247+0000","last_became_active":"2023-12-16T21:14:36.511047+0000","last_became_peered":"2023-12-16T21:14:36.511047+0000","last_unstale":"2023-12-16T21:14:37.494247+0000","last_undegraded":"2023-12-16T21:14:37.494247+0000","last_fullsized":"2023-12-16T21:14:37.494247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:00:03.357856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.509791+0000","last_change":"2023-12-16T21:14:35.746991+0000","last_active":"2023-12-16T21:14:36.509791+0000","last_peered":"2023-12-16T21:14:36.509791+0000","last_clean":"2023-12-16T21:14:36.509791+0000","last_became_active":"2023-12-16T21:14:35.745976+0000","last_became_peered":"2023-12-16T21:14:35.745976+0000","last_unstale":"2023-12-16T21:14:36.509791+0000","last_undegraded":"2023-12-16T21:14:36.509791+0000","last_fullsized":"2023-12-16T21:14:36.509791+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:56:09.194348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.514576+0000","last_change":"2023-12-16T21:14:36.509876+0000","last_active":"2023-12-16T21:14:36.514576+0000","last_peered":"2023-12-16T21:14:36.514576+0000","last_clean":"2023-12-16T21:14:36.514576+0000","last_became_active":"2023-12-16T21:14:36.508197+0000","last_became_peered":"2023-12-16T21:14:36.508197+0000","last_unstale":"2023-12-16T21:14:36.514576+0000","last_undegraded":"2023-12-16T21:14:36.514576+0000","last_fullsized":"2023-12-16T21:14:36.514576+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:28:38.343549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.494335+0000","last_change":"2023-12-16T21:14:35.751473+0000","last_active":"2023-12-16T21:14:37.494335+0000","last_peered":"2023-12-16T21:14:37.494335+0000","last_clean":"2023-12-16T21:14:37.494335+0000","last_became_active":"2023-12-16T21:14:35.749216+0000","last_became_peered":"2023-12-16T21:14:35.749216+0000","last_unstale":"2023-12-16T21:14:37.494335+0000","last_undegraded":"2023-12-16T21:14:37.494335+0000","last_fullsized":"2023-12-16T21:14:37.494335+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:57:38.342980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-16T21:14:37.499044+0000","last_change":"2023-12-16T21:14:36.505640+0000","last_active":"2023-12-16T21:14:37.499044+0000","last_peered":"2023-12-16T21:14:37.499044+0000","last_clean":"2023-12-16T21:14:37.499044+0000","last_became_active":"2023-12-16T21:14:36.505360+0000","last_became_peered":"2023-12-16T21:14:36.505360+0000","last_unstale":"2023-12-16T21:14:37.499044+0000","last_undegraded":"2023-12-16T21:14:37.499044+0000","last_fullsized":"2023-12-16T21:14:37.499044+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:35.729092+0000","last_clean_scrub_stamp":"2023-12-16T21:14:35.729092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:58:24.162996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.499735+0000","last_change":"2023-12-16T21:14:35.748772+0000","last_active":"2023-12-16T21:14:36.499735+0000","last_peered":"2023-12-16T21:14:36.499735+0000","last_clean":"2023-12-16T21:14:36.499735+0000","last_became_active":"2023-12-16T21:14:35.748435+0000","last_became_peered":"2023-12-16T21:14:35.748435+0000","last_unstale":"2023-12-16T21:14:36.499735+0000","last_undegraded":"2023-12-16T21:14:36.499735+0000","last_fullsized":"2023-12-16T21:14:36.499735+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:14:34.723664+0000","last_clean_scrub_stamp":"2023-12-16T21:14:34.723664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:18:33.626710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":176,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-12-16T21:14:36.498437+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:14:36.498437+0000","last_peered":"2023-12-16T21:14:36.498437+0000","last_clean":"2023-12-16T21:14:36.498437+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:14:36.498437+0000","last_undegraded":"2023-12-16T21:14:36.498437+0000","last_fullsized":"2023-12-16T21:14:36.498437+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":970752,"data_stored":735413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.51600000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.5},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.42999999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.45600000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:14:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58499999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.62},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.56100000000000005}]},{"osd":2,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60799999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54700000000000004}]},{"osd":4,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.63800000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.38700000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.42099999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":151927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.433},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.499}]},{"osd":1,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":5.3639999999999999,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.443},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.54800000000000004}]},{"osd":4,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53600000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.45900000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.57499999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":145059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55400000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54000000000000004}]},{"osd":5,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.63100000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.68100000000000005}]},{"osd":7,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61899999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":389120,"data_stored":146589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.38500000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.329,"5min":1.329,"15min":1.329},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.266,"5min":1.266,"15min":1.266},"last":0.42399999999999999}]},{"osd":2,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.371}]},{"osd":4,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.55400000000000005}]},{"osd":7,"last update":"Sat Dec 16 21:14:22 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.622},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.60499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":376832,"data_stored":145025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.246},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.318}]},{"osd":1,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.36899999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.35099999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.501},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.64100000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51000000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:14:25 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.67400000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31564,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340276,"statfs":{"total":96636764160,"available":96604442624,"internally_reserved":0,"allocated":389120,"data_stored":146589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.34499999999999997}]},{"osd":2,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.32500000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.62}]},{"osd":5,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.504}]},{"osd":6,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.54700000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:14:24 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.52100000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32132,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339708,"statfs":{"total":96636764160,"available":96603860992,"internally_reserved":0,"allocated":966656,"data_stored":735323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.311},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.32900000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.47699999999999998}]},{"osd":5,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Dec 16 21:14:27 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:14:40.103 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-16T21:14:40.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-16T21:14:40.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.250+0000 7f82bc385640 1 Processor -- start 2023-12-16T21:14:40.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.250+0000 7f82bc385640 1 -- start start 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.251+0000 7f82bc385640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 0x7f82b4071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.251+0000 7f82bc385640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b415aea0 con 0x7f82b415e300 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.251+0000 7f82bc385640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b4072090 con 0x7f82b40b0bf0 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.251+0000 7f82bc385640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b415b790 con 0x7f82b415b330 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.251+0000 7f82ba121640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f82b415e300 0x7f82b415e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36624/0 (socket says 172.21.15.46:36624) 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.251+0000 7f82ba121640 1 -- 172.21.15.46:0/1870398155 learned_addr learned my addr 172.21.15.46:0/1870398155 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b9920640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 0x7f82b4071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54581541 0 0) 0x7f82b415aea0 con 0x7f82b415e300 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c003520 con 0x7f82b415e300 2023-12-16T21:14:40.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710148960 0 0) 0x7f829c003520 con 0x7f82b415e300 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 msgr2=0x7f82b4071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 0x7f82b4071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 >> v1:172.21.15.121:6789/0 conn(0x7f82b40b0bf0 legacy=0x7f82b40ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4072370 con 0x7f82b415e300 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f82ac001670 con 0x7f82b415e300 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 >> v1:172.21.15.46:6789/0 conn(0x7f82b415e300 legacy=0x7f82b415e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f829c0039a0 0x7f829c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 0x7f829c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f829c004d00 0x7f829c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b415b790 con 0x7f829c0039a0 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c00db90 con 0x7f829c004d00 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829c00dd50 con 0x7f82b415b330 2023-12-16T21:14:40.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba121640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f829c0039a0 0x7f829c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82b9920640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 0x7f829c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba922640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f829c004d00 0x7f829c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba121640 1 -- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 msgr2=0x7f829c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba121640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f82b415b330 0x7f829c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba121640 1 -- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f829c004d00 msgr2=0x7f829c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba121640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f829c004d00 0x7f829c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.252+0000 7f82ba121640 1 -- 172.21.15.46:0/1870398155 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c003520 con 0x7f829c0039a0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.253+0000 7f82ba121640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f829c0039a0 0x7f829c003d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00c210 tx=0x7f829c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f766b5ad12c53e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.253+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac005ad0 con 0x7f829c0039a0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.253+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ac005c50 con 0x7f829c0039a0 2023-12-16T21:14:40.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.253+0000 7f82b911f640 1 -- 172.21.15.46:0/1870398155 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac02c830 con 0x7f829c0039a0 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82bc385640 1 -- 172.21.15.46:0/1870398155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f829c0039a0 msgr2=0x7f829c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82bc385640 1 --2- 172.21.15.46:0/1870398155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f829c0039a0 0x7f829c003d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f82ac00c210 tx=0x7f829c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82ba922640 1 -- 172.21.15.46:0/1870398155 reap_dead start 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82bc385640 1 -- 172.21.15.46:0/1870398155 shutdown_connections 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82bc385640 1 -- 172.21.15.46:0/1870398155 >> 172.21.15.46:0/1870398155 conn(0x7f82b406d9c0 msgr2=0x7f82b406a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82bc385640 1 -- 172.21.15.46:0/1870398155 shutdown_connections 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.254+0000 7f82bc385640 1 -- 172.21.15.46:0/1870398155 wait complete. 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.255+0000 7f82bc385640 1 Processor -- start 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.255+0000 7f82bc385640 1 -- start start 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82bc385640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829c0039a0 0x7f82b41593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82bc385640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 0x7f82b4141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82bc385640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82b4137380 0x7f82b4137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82bc385640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4137d40 con 0x7f82b40afcc0 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82bc385640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4138fe0 con 0x7f82b4137380 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82bc385640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4159980 con 0x7f829c0039a0 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 0x7f82b4141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 0x7f82b4141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41778/0 (socket says 172.21.15.46:41778) 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 -- 172.21.15.46:0/2068662816 learned_addr learned my addr 172.21.15.46:0/2068662816 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82ba922640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82b4137380 0x7f82b4137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82ba121640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829c0039a0 0x7f82b41593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 -- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829c0039a0 msgr2=0x7f82b41593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829c0039a0 0x7f82b41593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 -- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82b4137380 msgr2=0x7f82b4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82b4137380 0x7f82b4137800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.256+0000 7f82b9920640 1 -- 172.21.15.46:0/2068662816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b4159c60 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.257+0000 7f82b9920640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 0x7f82b4141320 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f82a80119d0 tx=0x7f82a8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.257+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a800fb50 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.257+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a800fcb0 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.257+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b4159f40 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.257+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a8014440 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.257+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b4141860 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.258+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b4138e20 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.258+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f82a80145a0 con 0x7f82b40afcc0 2023-12-16T21:14:40.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.259+0000 7f82b2ffd640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82a006ae40 0x7f82a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.259+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82a8089cd0 con 0x7f82b40afcc0 2023-12-16T21:14:40.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.259+0000 7f82ba121640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82a006ae40 0x7f82a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.260+0000 7f82ba121640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82a006ae40 0x7f82a006d300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82ac034000 tx=0x7f82ac009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:40.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.263+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f82b4138e20 con 0x7f82b40afcc0 2023-12-16T21:14:40.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.505+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f82b4138e20 con 0x7f82b40afcc0 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.507+0000 7f82b2ffd640 1 -- 172.21.15.46:0/2068662816 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f82b4138e20 con 0x7f82b40afcc0 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-12-16T21:14:40.508 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82a006ae40 msgr2=0x7f82a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82a006ae40 0x7f82a006d300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f82ac034000 tx=0x7f82ac009030 comp rx=0 tx=0).stop 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 msgr2=0x7f82b4141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 0x7f82b4141320 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f82a80119d0 tx=0x7f82a8011e90 comp rx=0 tx=0).stop 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 shutdown_connections 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f82a006ae40 0x7f82a006d300 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82b40afcc0 0x7f82b4141320 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f82b4137380 0x7f82b4137800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 --2- 172.21.15.46:0/2068662816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f829c0039a0 0x7f82b41593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 >> 172.21.15.46:0/2068662816 conn(0x7f82b406d9c0 msgr2=0x7f82b40ccdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:40.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 shutdown_connections 2023-12-16T21:14:40.512 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.510+0000 7f82bc385640 1 -- 172.21.15.46:0/2068662816 wait complete. 2023-12-16T21:14:40.522 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.627+0000 7fa57d533640 1 Processor -- start 2023-12-16T21:14:40.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.628+0000 7fa57d533640 1 -- start start 2023-12-16T21:14:40.629 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.628+0000 7fa57d533640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa57815d2d0 0x7fa5780b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.628+0000 7fa57d533640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5780ce650 0x7fa5780d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.628+0000 7fa57d533640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5781614e0 con 0x7fa578160f70 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.628+0000 7fa57d533640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5781617c0 con 0x7fa5780ce650 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.628+0000 7fa57d533640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa578161aa0 con 0x7fa57815d2d0 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5777fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5780ce650 0x7fa5780d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5767fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa578160f70 0x7fa5780cdf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60580/0 (socket says 172.21.15.46:60580) 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5767fc640 1 -- 172.21.15.46:0/866882518 learned_addr learned my addr 172.21.15.46:0/866882518 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:40.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa576ffd640 1 --2- 172.21.15.46:0/866882518 >> v2:172.21.15.121:3300/0 conn(0x7fa57815d2d0 0x7fa5780b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5777fe640 1 -- 172.21.15.46:0/866882518 >> v2:172.21.15.121:3300/0 conn(0x7fa57815d2d0 msgr2=0x7fa5780b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:40.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5777fe640 1 --2- 172.21.15.46:0/866882518 >> v2:172.21.15.121:3300/0 conn(0x7fa57815d2d0 0x7fa5780b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5777fe640 1 -- 172.21.15.46:0/866882518 >> v1:172.21.15.46:6789/0 conn(0x7fa578160f70 legacy=0x7fa5780cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa5777fe640 1 -- 172.21.15.46:0/866882518 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578161d80 con 0x7fa5780ce650 2023-12-16T21:14:40.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.629+0000 7fa576ffd640 1 --2- 172.21.15.46:0/866882518 >> v2:172.21.15.121:3300/0 conn(0x7fa57815d2d0 0x7fa5780b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa5777fe640 1 --2- 172.21.15.46:0/866882518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780d1af0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa5780b5db0 tx=0x7fa56802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8d7273e077d6eb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa575ffb640 1 -- 172.21.15.46:0/866882518 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56802cc80 con 0x7fa5780ce650 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa575ffb640 1 -- 172.21.15.46:0/866882518 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56802cde0 con 0x7fa5780ce650 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa575ffb640 1 -- 172.21.15.46:0/866882518 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568034660 con 0x7fa5780ce650 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 -- 172.21.15.46:0/866882518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 msgr2=0x7fa5780d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 --2- 172.21.15.46:0/866882518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780d1af0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa5780b5db0 tx=0x7fa56802bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 -- 172.21.15.46:0/866882518 shutdown_connections 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 --2- 172.21.15.46:0/866882518 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780d1af0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 --2- 172.21.15.46:0/866882518 >> v2:172.21.15.121:3300/0 conn(0x7fa57815d2d0 0x7fa5780b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 -- 172.21.15.46:0/866882518 >> 172.21.15.46:0/866882518 conn(0x7fa578072230 msgr2=0x7fa5780cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.630+0000 7fa57d533640 1 -- 172.21.15.46:0/866882518 shutdown_connections 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.631+0000 7fa57d533640 1 -- 172.21.15.46:0/866882518 wait complete. 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.632+0000 7fa57d533640 1 Processor -- start 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.632+0000 7fa57d533640 1 -- start start 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57815d2d0 0x7fa5780c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa578160f70 0x7fa5781534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780c7580 con 0x7fa5780ce650 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780c7860 con 0x7fa57815d2d0 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780b3dc0 con 0x7fa578160f70 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780c6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46678/0 (socket says 172.21.15.46:46678) 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 -- 172.21.15.46:0/3881407145 learned_addr learned my addr 172.21.15.46:0/3881407145 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa5767fc640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57815d2d0 0x7fa5780c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa5777fe640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa578160f70 0x7fa5781534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 -- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa578160f70 msgr2=0x7fa5781534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa578160f70 0x7fa5781534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 -- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57815d2d0 msgr2=0x7fa5780c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57815d2d0 0x7fa5780c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 -- 172.21.15.46:0/3881407145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa568009a70 con 0x7fa5780ce650 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa576ffd640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780c6d30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa57000c7f0 tx=0x7fa57000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570007a30 con 0x7fa5780ce650 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa570007b90 con 0x7fa5780ce650 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5780b40a0 con 0x7fa5780ce650 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.633+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57000ce40 con 0x7fa5780ce650 2023-12-16T21:14:40.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.634+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5780c9790 con 0x7fa5780ce650 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.634+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5780c7aa0 con 0x7fa5780ce650 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.635+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa5700113f0 con 0x7fa5780ce650 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.635+0000 7fa56f7fe640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa55806ae70 0x7fa55806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.635+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa570089740 con 0x7fa5780ce650 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.635+0000 7fa5767fc640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa55806ae70 0x7fa55806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.636+0000 7fa5767fc640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa55806ae70 0x7fa55806d330 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa5780b75f0 tx=0x7fa560009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:40.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.639+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa5700548a0 con 0x7fa5780ce650 2023-12-16T21:14:40.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.852+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5780b8870 con 0x7fa5780ce650 2023-12-16T21:14:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.853+0000 7fa56f7fe640 1 -- 172.21.15.46:0/3881407145 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4940 (secure 0 0 0) 0x7fa570057ec0 con 0x7fa5780ce650 2023-12-16T21:14:40.854 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:40.854 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:38.102533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:14:40.854 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2023-12-16T21:14:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa55806ae70 msgr2=0x7fa55806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa55806ae70 0x7fa55806d330 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa5780b75f0 tx=0x7fa560009040 comp rx=0 tx=0).stop 2023-12-16T21:14:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 msgr2=0x7fa5780c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780c6d30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa57000c7f0 tx=0x7fa57000cbf0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 shutdown_connections 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa55806ae70 0x7fa55806d330 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa578160f70 0x7fa5781534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57815d2d0 0x7fa5780c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 --2- 172.21.15.46:0/3881407145 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5780ce650 0x7fa5780c6d30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 >> 172.21.15.46:0/3881407145 conn(0x7fa578072230 msgr2=0x7fa5780b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 shutdown_connections 2023-12-16T21:14:40.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:40.855+0000 7fa57d533640 1 -- 172.21.15.46:0/3881407145 wait complete. 2023-12-16T21:14:41.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 13, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-16T21:14:36.098017+0000', 'modified': '2023-12-16T21:14:38.102533+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5244}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5244': {'gid': 5244, 'name': 'a', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.46:6835/3134379513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6833', 'nonce': 3134379513}, {'type': 'v1', 'addr': '172.21.15.46:6835', 'nonce': 3134379513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-16T21:14:41.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-16T21:14:41.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-16T21:14:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.379+0000 7f337d14c640 1 Processor -- start 2023-12-16T21:14:41.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.379+0000 7f337d14c640 1 -- start start 2023-12-16T21:14:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f337d14c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f337d14c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33780b2b10 con 0x7f33780d2910 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f337d14c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33780d1590 con 0x7f33780d2d90 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f337d14c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33780d1870 con 0x7f337815d330 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f337659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f33780d2910 0x7f33780d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60586/0 (socket says 172.21.15.46:60586) 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f337659c640 1 -- 172.21.15.46:0/777754071 learned_addr learned my addr 172.21.15.46:0/777754071 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.380+0000 7f3376d9d640 1 --2- 172.21.15.46:0/777754071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338115055 0 0) 0x7f33780b2b10 con 0x7f33780d2910 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3364003520 con 0x7f33780d2910 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177051110 0 0) 0x7f33780d1590 con 0x7f33780d2d90 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33780b2b10 con 0x7f33780d2d90 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3376d9d640 1 -- 172.21.15.46:0/777754071 >> v1:172.21.15.121:6789/0 conn(0x7f33780d2d90 legacy=0x7f33780b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3376d9d640 1 -- 172.21.15.46:0/777754071 >> v1:172.21.15.46:6789/0 conn(0x7f33780d2910 legacy=0x7f33780d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.382 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.381+0000 7f3376d9d640 1 -- 172.21.15.46:0/777754071 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33780d1b50 con 0x7f337815d330 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f3376d9d640 1 --2- 172.21.15.46:0/777754071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780b8b80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3368009dc0 tx=0x7f336802be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70452ba9595816e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336802cc40 con 0x7f337815d330 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336802cda0 con 0x7f337815d330 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f3375d9b640 1 -- 172.21.15.46:0/777754071 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368034660 con 0x7f337815d330 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 -- 172.21.15.46:0/777754071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 msgr2=0x7f33780b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 --2- 172.21.15.46:0/777754071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780b8b80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3368009dc0 tx=0x7f336802be90 comp rx=0 tx=0).stop 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 -- 172.21.15.46:0/777754071 shutdown_connections 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 --2- 172.21.15.46:0/777754071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780b8b80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 -- 172.21.15.46:0/777754071 >> 172.21.15.46:0/777754071 conn(0x7f3378072020 msgr2=0x7f3378072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 -- 172.21.15.46:0/777754071 shutdown_connections 2023-12-16T21:14:41.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.382+0000 7f337d14c640 1 -- 172.21.15.46:0/777754071 wait complete. 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 Processor -- start 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 -- start start 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 0x7f3378152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33780d2d90 0x7f33780ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33780cc8c0 con 0x7f33780d2910 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337815f5b0 con 0x7f33780d2d90 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.384+0000 7f337d14c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337815f890 con 0x7f337815d330 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 0x7f3378152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 0x7f3378152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46694/0 (socket says 172.21.15.46:46694) 2023-12-16T21:14:41.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 -- 172.21.15.46:0/251720503 learned_addr learned my addr 172.21.15.46:0/251720503 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f337659c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33780d2d90 0x7f33780ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f337759e640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 -- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 msgr2=0x7f33780ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 -- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33780d2d90 msgr2=0x7f33780ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33780d2d90 0x7f33780ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 -- 172.21.15.46:0/251720503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3368009a70 con 0x7f33780d2910 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f337659c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33780d2d90 0x7f33780ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f3376d9d640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 0x7f3378152660 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3368000f80 tx=0x7f3368004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f336f7fe640 1 -- 172.21.15.46:0/251720503 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368040650 con 0x7f33780d2910 2023-12-16T21:14:41.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f336f7fe640 1 -- 172.21.15.46:0/251720503 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33680407b0 con 0x7f33780d2910 2023-12-16T21:14:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f336f7fe640 1 -- 172.21.15.46:0/251720503 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368040b60 con 0x7f33780d2910 2023-12-16T21:14:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337815fb70 con 0x7f33780d2910 2023-12-16T21:14:41.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.385+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3378160130 con 0x7f33780d2910 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.387+0000 7f336f7fe640 1 -- 172.21.15.46:0/251720503 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3368040cc0 con 0x7f33780d2910 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.387+0000 7f336f7fe640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f335806ae40 0x7f335806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.388+0000 7f336f7fe640 1 -- 172.21.15.46:0/251720503 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33680ac330 con 0x7f33780d2910 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.388+0000 7f337659c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f335806ae40 0x7f335806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.388+0000 7f337659c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f335806ae40 0x7f335806d300 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3360007fd0 tx=0x7f3360009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.388+0000 7f337fdbf740 1 Processor -- start 2023-12-16T21:14:41.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 -- start start 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631960 0x558001604b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 0x5580016346f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5580016050d0 0x558001634c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x558001578090 con 0x558001631d40 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x558001575780 con 0x558001631960 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x558001571070 con 0x5580016050d0 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 0x5580016346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 0x5580016346f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46702/0 (socket says 172.21.15.46:46702) 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337659c640 1 -- 172.21.15.46:0/2283958879 learned_addr learned my addr 172.21.15.46:0/2283958879 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f337759e640 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5580016050d0 0x558001634c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.389+0000 7f3376d9d640 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631960 0x558001604b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f337659c640 1 -- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5580016050d0 msgr2=0x558001634c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f337659c640 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5580016050d0 0x558001634c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f337659c640 1 -- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631960 msgr2=0x558001604b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f337659c640 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631960 0x558001604b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f337659c640 1 -- 172.21.15.46:0/2283958879 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558001533d00 con 0x558001631d40 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f337659c640 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 0x5580016346f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3360005910 tx=0x7f3360011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9fb2b81ca0e495 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f336cff9640 1 -- 172.21.15.46:0/2283958879 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360005e90 con 0x558001631d40 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f336cff9640 1 -- 172.21.15.46:0/2283958879 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336000e7b0 con 0x558001631d40 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.390+0000 7f336cff9640 1 -- 172.21.15.46:0/2283958879 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336000eb20 con 0x558001631d40 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 -- 172.21.15.46:0/2283958879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 msgr2=0x5580016346f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 0x5580016346f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3360005910 tx=0x7f3360011000 comp rx=0 tx=0).stop 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 -- 172.21.15.46:0/2283958879 shutdown_connections 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631d40 0x5580016346f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5580016050d0 0x558001634c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 --2- 172.21.15.46:0/2283958879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631960 0x558001604b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 -- 172.21.15.46:0/2283958879 >> 172.21.15.46:0/2283958879 conn(0x558001606e00 msgr2=0x55800164f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 -- 172.21.15.46:0/2283958879 shutdown_connections 2023-12-16T21:14:41.394 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 -- 172.21.15.46:0/2283958879 wait complete. 2023-12-16T21:14:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.391+0000 7f337fdbf740 1 -- start start 2023-12-16T21:14:41.395 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 Processor -- start 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x558001649350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x558001637140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x558001662b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800152ae70 con 0x5580016626d0 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800152ab10 con 0x558001631d40 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580014f02e0 con 0x558001631960 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337fdbf740 10 client.0 _ll_register_callbacks cb 0x558001282ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x558001662b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x558001662b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46714/0 (socket says 172.21.15.46:46714) 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337759e640 1 -- 172.21.15.46:0/768293745 learned_addr learned my addr 172.21.15.46:0/768293745 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:41.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f337659c640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x558001637140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.392+0000 7f3376d9d640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x558001649350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337759e640 1 -- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 msgr2=0x558001649350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337759e640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x558001649350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337759e640 1 -- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 msgr2=0x558001637140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337759e640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x558001637140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337759e640 1 -- 172.21.15.46:0/768293745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336000f040 con 0x5580016626d0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337759e640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x558001662b30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3370001e00 tx=0x7f3370011620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4cd0dac1148b48c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f3355ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f3355ffb640 1 -- 172.21.15.46:0/768293745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370011eb0 con 0x5580016626d0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f3355ffb640 1 -- 172.21.15.46:0/768293745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33700052e0 con 0x5580016626d0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558001569f00 con 0x5580016626d0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f3355ffb640 1 -- 172.21.15.46:0/768293745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33700054a0 con 0x5580016626d0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337fdbf740 10 client.5418 fetch_fsmap learned FSMap version 13 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x55800156f6f0 con 0x5580016626d0 2023-12-16T21:14:41.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f3355ffb640 1 -- 172.21.15.46:0/768293745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558001569f00 con 0x5580016626d0 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.393+0000 7f3355ffb640 1 -- 172.21.15.46:0/768293745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 13) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f33700035f0 con 0x5580016626d0 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f337fdbf740 10 client.5418 fetch_fsmap finished waiting for FSMap version 13 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f337fdbf740 10 client.5418 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f337fdbf740 20 client.5418 populate_metadata read hostname 'smithi046' 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f337fdbf740 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001620c30 0x558001621780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f337fdbf740 4 client.5418 mds_command: new command op to 5244 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 --> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558001569c20 con 0x558001620c30 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f3376d9d640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001620c30 0x558001621780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f3376d9d640 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001620c30 0x558001621780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37c544d3113e9a7f server_cookie=9c522ad5bbae4fc0 in_seq=0 out_seq=0 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f3355ffb640 10 client.5418 ms_handle_connect on v2:172.21.15.46:6833/3134379513 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f3355ffb640 1 -- 172.21.15.46:0/768293745 <== mds.0 v2:172.21.15.46:6833/3134379513 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558001569c20 con 0x558001620c30 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.394+0000 7f3355ffb640 10 client.5418 handle_command_reply: tid=0 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 client.5418 shutdown 2023-12-16T21:14:41.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 msgr2=0x558001662b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x558001662b30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3370001e00 tx=0x7f3370011620 comp rx=0 tx=0).stop 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 shutdown_connections 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001620c30 0x558001621780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x558001662b30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f3355ffb640 0 client.5418 ms_handle_reset on v2:172.21.15.46:6833/3134379513 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x558001637140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 --2- 172.21.15.46:0/768293745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x558001649350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 >> 172.21.15.46:0/768293745 conn(0x5580016050d0 msgr2=0x558001608920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 shutdown_connections 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 1 -- 172.21.15.46:0/768293745 wait complete. 2023-12-16T21:14:41.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.395+0000 7f337fdbf740 20 client.5418 trim_cache size 0 max 16384 2023-12-16T21:14:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 Processor -- start 2023-12-16T21:14:41.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 -- start start 2023-12-16T21:14:41.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 0x5580016523f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631d40 0x558001608460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580016626d0 0x5580016485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580016081c0 con 0x558001631960 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x558001636f50 con 0x5580016626d0 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.411+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x558001652930 con 0x558001631d40 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 0x5580016523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 0x5580016523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46726/0 (socket says 172.21.15.46:46726) 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 -- 172.21.15.46:0/472324114 learned_addr learned my addr 172.21.15.46:0/472324114 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 -- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631d40 msgr2=0x558001608460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631d40 0x558001608460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.413 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f337759e640 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580016626d0 0x5580016485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 -- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580016626d0 msgr2=0x5580016485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580016626d0 0x5580016485f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 -- 172.21.15.46:0/472324114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55800150cd70 con 0x558001631960 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f3376d9d640 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 0x5580016523f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f33680775a0 tx=0x7f33680ac040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e871397b24e723 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f336d7fa640 1 -- 172.21.15.46:0/472324114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336807b0a0 con 0x558001631960 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f336d7fa640 1 -- 172.21.15.46:0/472324114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336807f1f0 con 0x558001631960 2023-12-16T21:14:41.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.412+0000 7f336d7fa640 1 -- 172.21.15.46:0/472324114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33680833a0 con 0x558001631960 2023-12-16T21:14:41.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 -- 172.21.15.46:0/472324114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 msgr2=0x5580016523f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 0x5580016523f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f33680775a0 tx=0x7f33680ac040 comp rx=0 tx=0).stop 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 -- 172.21.15.46:0/472324114 shutdown_connections 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5580016626d0 0x5580016485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631d40 0x558001608460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 --2- 172.21.15.46:0/472324114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x558001631960 0x5580016523f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 -- 172.21.15.46:0/472324114 >> 172.21.15.46:0/472324114 conn(0x5580016050d0 msgr2=0x558001671cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 -- 172.21.15.46:0/472324114 shutdown_connections 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 -- 172.21.15.46:0/472324114 wait complete. 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 -- start start 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.413+0000 7f337fdbf740 1 Processor -- start 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x55800164a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x55800162f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x5580016a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55800161f130 con 0x5580016626d0 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x558001651ac0 con 0x558001631d40 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x55800161eca0 con 0x558001631960 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337fdbf740 10 client.0 _ll_register_callbacks cb 0x7f3360001550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x5580016a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x5580016a28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46732/0 (socket says 172.21.15.46:46732) 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 -- 172.21.15.46:0/1778638223 learned_addr learned my addr 172.21.15.46:0/1778638223 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337659c640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x55800162f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f3376d9d640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x55800164a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 -- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 msgr2=0x55800164a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x55800164a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 -- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 msgr2=0x55800162f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x55800162f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337759e640 1 -- 172.21.15.46:0/1778638223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3378160130 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.414+0000 7f337659c640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x55800162f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337759e640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x5580016a28a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3370001eb0 tx=0x7f3370001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1df559d5364819 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3355ffb640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3355ffb640 1 -- 172.21.15.46:0/1778638223 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3370007b00 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3355ffb640 1 -- 172.21.15.46:0/1778638223 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3370003d10 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f336000e7b0 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3355ffb640 1 -- 172.21.15.46:0/1778638223 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3370011b20 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 10 client.5424 fetch_fsmap learned FSMap version 13 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=13,monmap=2+}) v3 -- 0x558001532560 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3355ffb640 1 -- 172.21.15.46:0/1778638223 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336000e7b0 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3355ffb640 1 -- 172.21.15.46:0/1778638223 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 13) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3370015470 con 0x5580016626d0 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 10 client.5424 fetch_fsmap finished waiting for FSMap version 13 2023-12-16T21:14:41.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 10 client.5424 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 20 client.5424 populate_metadata read hostname 'smithi046' 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001650ee0 0x5580016bd010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 4 client.5424 mds_command: new command op to 5244 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 --> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5580012716c0 con 0x558001650ee0 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.415+0000 7f3376d9d640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001650ee0 0x5580016bd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f3355ffb640 10 client.5424 ms_handle_connect on v2:172.21.15.46:6833/3134379513 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f3376d9d640 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001650ee0 0x5580016bd010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96147653da7995c5 server_cookie=1e08e5113bca8c05 in_seq=0 out_seq=0 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f3355ffb640 1 -- 172.21.15.46:0/1778638223 <== mds.0 v2:172.21.15.46:6833/3134379513 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5580012716c0 con 0x558001650ee0 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f3355ffb640 10 client.5424 handle_command_reply: tid=0 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 client.5424 shutdown 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 msgr2=0x5580016a28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x5580016a28a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3370001eb0 tx=0x7f3370001150 comp rx=0 tx=0).stop 2023-12-16T21:14:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 shutdown_connections 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:6833/3134379513,v1:172.21.15.46:6835/3134379513] conn(0x558001650ee0 0x5580016bd010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580016626d0 0x5580016a28a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558001631d40 0x55800162f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f3355ffb640 0 client.5424 ms_handle_reset on v2:172.21.15.46:6833/3134379513 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 --2- 172.21.15.46:0/1778638223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x558001631960 0x55800164a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 >> 172.21.15.46:0/1778638223 conn(0x5580016050d0 msgr2=0x558001630220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 shutdown_connections 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 1 -- 172.21.15.46:0/1778638223 wait complete. 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.416+0000 7f337fdbf740 20 client.5424 trim_cache size 0 max 16384 2023-12-16T21:14:41.424 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "2397e6ee-7ffe-4438-b275-b4e10dea2d2f", 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5244, 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.6714734460000003, 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 13, 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 34, 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 34, 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 11.548134481 2023-12-16T21:14:41.425 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:14:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f335806ae40 msgr2=0x7f335806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f335806ae40 0x7f335806d300 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3360007fd0 tx=0x7f3360009040 comp rx=0 tx=0).stop 2023-12-16T21:14:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 msgr2=0x7f3378152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 0x7f3378152660 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3368000f80 tx=0x7f3368004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 shutdown_connections 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f335806ae40 0x7f335806d300 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f337815d330 0x7f33780ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33780d2d90 0x7f33780ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 --2- 172.21.15.46:0/251720503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f33780d2910 0x7f3378152660 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 >> 172.21.15.46:0/251720503 conn(0x7f3378072020 msgr2=0x7f33780b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 shutdown_connections 2023-12-16T21:14:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.417+0000 7f337d14c640 1 -- 172.21.15.46:0/251720503 wait complete. 2023-12-16T21:14:41.428 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:14:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.534+0000 7fb7f34ec640 1 Processor -- start 2023-12-16T21:14:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.534+0000 7fb7f34ec640 1 -- start start 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.535+0000 7fb7f34ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7ec0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.535+0000 7fb7f34ec640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec0b2b10 con 0x7fb7ec0d2910 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.535+0000 7fb7f34ec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec0d1590 con 0x7fb7ec15d330 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.535+0000 7fb7f34ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0d1870 con 0x7fb7ec0d3900 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.535+0000 7fb7f0a87640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb7ec0d2910 0x7fb7ec0d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60594/0 (socket says 172.21.15.46:60594) 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.535+0000 7fb7f0a87640 1 -- 172.21.15.46:0/1806459759 learned_addr learned my addr 172.21.15.46:0/1806459759 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7f1a89640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7ec0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816106237 0 0) 0x7fb7ec0b2b10 con 0x7fb7ec0d2910 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d8003520 con 0x7fb7ec0d2910 2023-12-16T21:14:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597811856 0 0) 0x7fb7ec0d1590 con 0x7fb7ec15d330 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7ec0b2b10 con 0x7fb7ec15d330 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2772316029 0 0) 0x7fb7d8003520 con 0x7fb7ec0d2910 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 msgr2=0x7fb7ec0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7ec0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 >> v1:172.21.15.121:6789/0 conn(0x7fb7ec15d330 legacy=0x7fb7ec0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0d1b50 con 0x7fb7ec0d2910 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fb7dc001670 con 0x7fb7ec0d2910 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7f1a89640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7ec0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 >> v1:172.21.15.46:6789/0 conn(0x7fb7ec0d2910 legacy=0x7fb7ec0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8003b30 0x7fb7d8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7d8004d00 0x7fb7d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0d1870 con 0x7fb7d8003b30 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d800ba60 con 0x7fb7d8004d00 2023-12-16T21:14:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d800bc20 con 0x7fb7ec0d3900 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7f1288640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8003b30 0x7fb7d8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7f0a87640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.536+0000 7fb7f1a89640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7d8004d00 0x7fb7d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7f1288640 1 -- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 msgr2=0x7fb7d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7f1288640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7ec0d3900 0x7fb7d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7f1288640 1 -- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7d8004d00 msgr2=0x7fb7d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7f1288640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7d8004d00 0x7fb7d8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7f1288640 1 -- 172.21.15.46:0/1806459759 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e4009a70 con 0x7fb7d8003b30 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7f1288640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8003b30 0x7fb7d8003f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80018d0 tx=0x7fb7d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47a0b133ef54684 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e80052d0 con 0x7fb7d8003b30 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e8005430 con 0x7fb7d8003b30 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.537+0000 7fb7e3fff640 1 -- 172.21.15.46:0/1806459759 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8003810 con 0x7fb7d8003b30 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f34ec640 1 -- 172.21.15.46:0/1806459759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8003b30 msgr2=0x7fb7d8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/1806459759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7d8003b30 0x7fb7d8003f30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80018d0 tx=0x7fb7d80057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f1a89640 1 -- 172.21.15.46:0/1806459759 reap_dead start 2023-12-16T21:14:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f34ec640 1 -- 172.21.15.46:0/1806459759 shutdown_connections 2023-12-16T21:14:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f34ec640 1 -- 172.21.15.46:0/1806459759 >> 172.21.15.46:0/1806459759 conn(0x7fb7ec072020 msgr2=0x7fb7ec072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f34ec640 1 -- 172.21.15.46:0/1806459759 shutdown_connections 2023-12-16T21:14:41.542 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.538+0000 7fb7f34ec640 1 -- 172.21.15.46:0/1806459759 wait complete. 2023-12-16T21:14:41.544 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.539+0000 7fb7f34ec640 1 Processor -- start 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.539+0000 7fb7f34ec640 1 -- start start 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f34ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7d8003b30 0x7fb7ec14da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f34ec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7ec0b3680 0x7fb7ec1f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f34ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 0x7fb7ec0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f34ec640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec14df70 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f34ec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0b2ea0 con 0x7fb7ec0b3680 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f34ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1f38f0 con 0x7fb7d8003b30 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 0x7fb7ec0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 0x7fb7ec0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46746/0 (socket says 172.21.15.46:46746) 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 -- 172.21.15.46:0/3549967788 learned_addr learned my addr 172.21.15.46:0/3549967788 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f0a87640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7ec0b3680 0x7fb7ec1f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.545 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1288640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7d8003b30 0x7fb7ec14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 -- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7d8003b30 msgr2=0x7fb7ec14da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7d8003b30 0x7fb7ec14da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 -- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7ec0b3680 msgr2=0x7fb7ec1f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7ec0b3680 0x7fb7ec1f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1a89640 1 -- 172.21.15.46:0/3549967788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec1f3bd0 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.540+0000 7fb7f1288640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7d8003b30 0x7fb7ec14da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.541+0000 7fb7f1a89640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 0x7fb7ec0cc500 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec08c510 tx=0x7fb7e400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.541+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400e040 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.541+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e400f040 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.541+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4003bd0 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.541+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec0b3160 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.541+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec1f3320 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.542+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec0b3350 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.542+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb7e4015430 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.543+0000 7fb7e1ffb640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb7d006ae70 0x7fb7d006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:41.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.543+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7e4089f00 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.543+0000 7fb7f1288640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb7d006ae70 0x7fb7d006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:41.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.544+0000 7fb7f1288640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb7d006ae70 0x7fb7d006d330 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8034000 tx=0x7fb7e8031000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:41.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.547+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb7e4055040 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.725 INFO:tasks.ceph.mds.a.smithi046.stderr:2023-12-16T21:14:41.724+0000 7fccbfcae640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:14:41.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.761+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb7ec069400 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.761+0000 7fb7e1ffb640 1 -- 172.21.15.46:0/3549967788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb7e4058660 con 0x7fb7ec0b3b00 2023-12-16T21:14:41.762 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb7d006ae70 msgr2=0x7fb7d006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb7d006ae70 0x7fb7d006d330 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8034000 tx=0x7fb7e8031000 comp rx=0 tx=0).stop 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 msgr2=0x7fb7ec0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 0x7fb7ec0cc500 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec08c510 tx=0x7fb7e400f580 comp rx=0 tx=0).stop 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 shutdown_connections 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb7d006ae70 0x7fb7d006d330 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7ec0b3b00 0x7fb7ec0cc500 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7ec0b3680 0x7fb7ec1f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 --2- 172.21.15.46:0/3549967788 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7d8003b30 0x7fb7ec14da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:41.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 >> 172.21.15.46:0/3549967788 conn(0x7fb7ec072020 msgr2=0x7fb7ec0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:41.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 shutdown_connections 2023-12-16T21:14:41.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:41.763+0000 7fb7f34ec640 1 -- 172.21.15.46:0/3549967788 wait complete. 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:14:41.775 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:14:41.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:41.775 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:14:41.791 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:14:41.791 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:14:41.791 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:14:41.791 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.791 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42603sec preferred_lft 42603sec 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:14:41.792 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::20ee:6cff:fee1:5615/64 scope link 2023-12-16T21:14:41.793 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:41.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:41.794 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:14:41.861 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:14:41.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:41.863 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:14:41.919 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:14:41.920 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:14:41.920 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:14:41.920 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:14:41.920 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:14:41.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:41.920 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:14:41.920 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:14:41.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:14:41.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:14:41.921 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:14:41.921 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:14:42.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.027 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:14:42.042 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:14:42.042 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:14:42.042 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:14:42.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.043 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:14:42.114 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:14:42.114 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:14:42.114 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:42.169 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:14:42.170 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:14:42.170 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:42.224 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:14:42.296 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:14:42.353 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:14:42.354 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:42.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.356 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:14:42.409 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:14:42.409 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:14:42.452 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:14:42.483 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:14:42.539 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:14:42.539 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:14:42.540 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:42.541 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:42.542 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:42.542 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:42.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.542 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:14:42.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:14:42.546+0000 7f8a8b741480 -1 init, newargv = 0x55fe158333a0 newargc=15 2023-12-16T21:14:42.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[104135]: starting ceph client 2023-12-16T21:14:42.551 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[104135]: starting fuse 2023-12-16T21:14:42.562 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:14:42.562 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:14:42.563 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:14:42.564 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:14:42.687 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.104135.asok 2023-12-16T21:14:42.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:14:42.697 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.104135.asok 2023-12-16T21:14:42.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.698 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.104135.asok status 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:14:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "104135", 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5439, 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5439 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:14:42.841 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2458542257 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2458542257 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5439 192.168.0.1:0/2458542257", 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2458542257", 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 13, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 34, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:14:42.842 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:14:42.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.849 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:14:42.849 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:42.864 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:14:42.865 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:42.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:42.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:42.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2023-12-16T21:14:43.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.087+0000 7f2615783640 1 Processor -- start 2023-12-16T21:14:43.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.087+0000 7f2615783640 1 -- start start 2023-12-16T21:14:43.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f2615783640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 0x7f26100b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f2615783640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261015e2a0 con 0x7f261015abf0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f2615783640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261015b240 con 0x7f261015e730 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f2615783640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261015b520 con 0x7f2610143490 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f260f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f261015abf0 0x7f26100b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60602/0 (socket says 172.21.15.46:60602) 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f260f7fe640 1 -- 172.21.15.46:0/3086246791 learned_addr learned my addr 172.21.15.46:0/3086246791 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f260effd640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 0x7f26100b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737766199 0 0) 0x7f261015e2a0 con 0x7f261015abf0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.088+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25fc003520 con 0x7f261015abf0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709324550 0 0) 0x7f25fc003520 con 0x7f261015abf0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 msgr2=0x7f26100b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 0x7f26100b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 >> v1:172.21.15.121:6789/0 conn(0x7f261015e730 legacy=0x7f261015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261015dc50 con 0x7f261015abf0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f26080019f0 con 0x7f261015abf0 2023-12-16T21:14:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 >> v1:172.21.15.46:6789/0 conn(0x7f261015abf0 legacy=0x7f26100b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f25fc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 0x7f25fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25fc004d00 0x7f25fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261015b520 con 0x7f25fc0039a0 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc00ba60 con 0x7f25fc004d00 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc00de30 con 0x7f2610143490 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260effd640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f25fc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f2607fff640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 0x7f25fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260f7fe640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25fc004d00 0x7f25fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260effd640 1 -- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 msgr2=0x7f25fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260effd640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2610143490 0x7f25fc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260effd640 1 -- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25fc004d00 msgr2=0x7f25fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260effd640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25fc004d00 0x7f25fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260effd640 1 -- 172.21.15.46:0/3086246791 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc003520 con 0x7f25fc0039a0 2023-12-16T21:14:43.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.089+0000 7f260f7fe640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25fc004d00 0x7f25fc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.090+0000 7f260effd640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f25fc003dc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2600009a90 tx=0x7f25fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c9d967f899c775 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.090+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260002c680 con 0x7f25fc0039a0 2023-12-16T21:14:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.090+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260002c7e0 con 0x7f25fc0039a0 2023-12-16T21:14:43.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.090+0000 7f260e7fc640 1 -- 172.21.15.46:0/3086246791 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260002cb70 con 0x7f25fc0039a0 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.090+0000 7f2615783640 1 -- 172.21.15.46:0/3086246791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 msgr2=0x7f25fc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.090+0000 7f2615783640 1 --2- 172.21.15.46:0/3086246791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f25fc003dc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2600009a90 tx=0x7f25fc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.091+0000 7f260f7fe640 1 -- 172.21.15.46:0/3086246791 reap_dead start 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.091+0000 7f2615783640 1 -- 172.21.15.46:0/3086246791 shutdown_connections 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.091+0000 7f2615783640 1 -- 172.21.15.46:0/3086246791 >> 172.21.15.46:0/3086246791 conn(0x7f261006a4d0 msgr2=0x7f261006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.091+0000 7f2615783640 1 -- 172.21.15.46:0/3086246791 shutdown_connections 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.091+0000 7f2615783640 1 -- 172.21.15.46:0/3086246791 wait complete. 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.092+0000 7f2615783640 1 Processor -- start 2023-12-16T21:14:43.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.092+0000 7f2615783640 1 -- start start 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.092+0000 7f2615783640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f26100ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f261013ff10 0x7f2610140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26101592a0 0x7f2610159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2610141230 con 0x7f25fc0039a0 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2610159c60 con 0x7f261013ff10 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2610140990 con 0x7f26101592a0 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f26100ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f26100ab0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46756/0 (socket says 172.21.15.46:46756) 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 -- 172.21.15.46:0/4004028555 learned_addr learned my addr 172.21.15.46:0/4004028555 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260f7fe640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26101592a0 0x7f2610159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 -- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26101592a0 msgr2=0x7f2610159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26101592a0 0x7f2610159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 -- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f261013ff10 msgr2=0x7f2610140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f261013ff10 0x7f2610140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 -- 172.21.15.46:0/4004028555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2610140c70 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f260effd640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f26100ab0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f260002c220 tx=0x7f2600004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260002dea0 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2600003830 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2600003be0 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2610140f50 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.093+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261013ef80 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.094+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26100b4380 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.095+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2600003d40 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.095+0000 7f26077fe640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f25f006ae70 0x7f25f006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.095+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f26000ab3f0 con 0x7f25fc0039a0 2023-12-16T21:14:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.096+0000 7f2607fff640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f25f006ae70 0x7f25f006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.096+0000 7f2607fff640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f25f006ae70 0x7f25f006d330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f25f8000c00 tx=0x7f25f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.100+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2600076530 con 0x7f25fc0039a0 2023-12-16T21:14:43.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.275+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) v1 -- 0x7f26100a8690 con 0x7f25fc0039a0 2023-12-16T21:14:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.770+0000 7f26077fe640 1 -- 172.21.15.46:0/4004028555 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v35) v1 ==== 124+0+0 (secure 0 0 0) 0x7f26100a8690 con 0x7f25fc0039a0 2023-12-16T21:14:43.771 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'data_new' created 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f25f006ae70 msgr2=0x7f25f006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f25f006ae70 0x7f25f006d330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f25f8000c00 tx=0x7f25f8009040 comp rx=0 tx=0).stop 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 msgr2=0x7f26100ab0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f26100ab0e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f260002c220 tx=0x7f2600004730 comp rx=0 tx=0).stop 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 shutdown_connections 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f25f006ae70 0x7f25f006d330 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f26101592a0 0x7f2610159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f261013ff10 0x7f2610140390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 --2- 172.21.15.46:0/4004028555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f25fc0039a0 0x7f26100ab0e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 >> 172.21.15.46:0/4004028555 conn(0x7f261006a4d0 msgr2=0x7f261015d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 shutdown_connections 2023-12-16T21:14:43.775 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.773+0000 7f2615783640 1 -- 172.21.15.46:0/4004028555 wait complete. 2023-12-16T21:14:43.790 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2023-12-16T21:14:43.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.901+0000 7f4141800640 1 Processor -- start 2023-12-16T21:14:43.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.901+0000 7f4141800640 1 -- start start 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4141800640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f413c15e450 0x7f413c06a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4141800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f413c06aa10 0x7f413c0ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4141800640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413c15d710 con 0x7f413c157990 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4141800640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c0b46d0 con 0x7f413c15e450 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4141800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413c15e030 con 0x7f413c06aa10 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f413c15e450 0x7f413c06a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f413c15e450 0x7f413c06a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46772/0 (socket says 172.21.15.46:46772) 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f413affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f413c157990 0x7f413c0b3970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60606/0 (socket says 172.21.15.46:60606) 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 -- 172.21.15.46:0/1709037264 learned_addr learned my addr 172.21.15.46:0/1709037264 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:43.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f413b7fe640 1 --2- 172.21.15.46:0/1709037264 >> v2:172.21.15.121:3300/0 conn(0x7f413c06aa10 0x7f413c0ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 -- 172.21.15.46:0/1709037264 >> v2:172.21.15.121:3300/0 conn(0x7f413c06aa10 msgr2=0x7f413c0ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 --2- 172.21.15.46:0/1709037264 >> v2:172.21.15.121:3300/0 conn(0x7f413c06aa10 0x7f413c0ad420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 -- 172.21.15.46:0/1709037264 >> v1:172.21.15.46:6789/0 conn(0x7f413c157990 legacy=0x7f413c0b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.902+0000 7f4133fff640 1 -- 172.21.15.46:0/1709037264 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413c06ae40 con 0x7f413c15e450 2023-12-16T21:14:43.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.903+0000 7f4133fff640 1 --2- 172.21.15.46:0/1709037264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c15e450 0x7f413c06a4d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f413c15d8a0 tx=0x7f412402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1af27763dd67354 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:43.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.903+0000 7f413a7fc640 1 -- 172.21.15.46:0/1709037264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41240317e0 con 0x7f413c15e450 2023-12-16T21:14:43.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.903+0000 7f413a7fc640 1 -- 172.21.15.46:0/1709037264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4124031940 con 0x7f413c15e450 2023-12-16T21:14:43.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.903+0000 7f413a7fc640 1 -- 172.21.15.46:0/1709037264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124031cd0 con 0x7f413c15e450 2023-12-16T21:14:43.905 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 -- 172.21.15.46:0/1709037264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c15e450 msgr2=0x7f413c06a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 --2- 172.21.15.46:0/1709037264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c15e450 0x7f413c06a4d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f413c15d8a0 tx=0x7f412402b880 comp rx=0 tx=0).stop 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 -- 172.21.15.46:0/1709037264 shutdown_connections 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 --2- 172.21.15.46:0/1709037264 >> v2:172.21.15.121:3300/0 conn(0x7f413c06aa10 0x7f413c0ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 --2- 172.21.15.46:0/1709037264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c15e450 0x7f413c06a4d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 -- 172.21.15.46:0/1709037264 >> 172.21.15.46:0/1709037264 conn(0x7f413c0b1ad0 msgr2=0x7f413c0b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 -- 172.21.15.46:0/1709037264 shutdown_connections 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.904+0000 7f4141800640 1 -- 172.21.15.46:0/1709037264 wait complete. 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.905+0000 7f4141800640 1 Processor -- start 2023-12-16T21:14:43.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.905+0000 7f4141800640 1 -- start start 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c06aa10 0x7f413c142990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 0x7f413c1f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f413c15e450 0x7f413c1f6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0b1640 con 0x7f413c157990 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0b09c0 con 0x7f413c06aa10 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413c0b0ca0 con 0x7f413c15e450 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 0x7f413c1f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 0x7f413c1f62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46786/0 (socket says 172.21.15.46:46786) 2023-12-16T21:14:43.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 -- 172.21.15.46:0/2617354805 learned_addr learned my addr 172.21.15.46:0/2617354805 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f413affd640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c06aa10 0x7f413c142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f413b7fe640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f413c15e450 0x7f413c1f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 -- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f413c15e450 msgr2=0x7f413c1f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f413c15e450 0x7f413c1f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 -- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c06aa10 msgr2=0x7f413c142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c06aa10 0x7f413c142990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 -- 172.21.15.46:0/2617354805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240094d0 con 0x7f413c157990 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f413affd640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c06aa10 0x7f413c142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:43.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f413b7fe640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f413c15e450 0x7f413c1f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4133fff640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 0x7f413c1f62d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4124000c00 tx=0x7f41240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124032880 con 0x7f413c157990 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4124032a00 con 0x7f413c157990 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413c0b0f80 con 0x7f413c157990 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.906+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4124032db0 con 0x7f413c157990 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.907+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413c0afbf0 con 0x7f413c157990 2023-12-16T21:14:43.910 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.907+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f413c069320 con 0x7f413c157990 2023-12-16T21:14:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.908+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f412404e450 con 0x7f413c157990 2023-12-16T21:14:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.908+0000 7f41337fe640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f411c06ae40 0x7f411c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.909+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6072+0+0 (secure 0 0 0) 0x7f41240ab9f0 con 0x7f413c157990 2023-12-16T21:14:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.909+0000 7f413affd640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f411c06ae40 0x7f411c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:43.911 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.909+0000 7f413affd640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f411c06ae40 0x7f411c06d300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f412c000f60 tx=0x7f412c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:43.913 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:43.913+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f41240769e0 con 0x7f413c157990 2023-12-16T21:14:44.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.130+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) v1 -- 0x7f413c06b310 con 0x7f413c157990 2023-12-16T21:14:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.788+0000 7f41337fe640 1 -- 172.21.15.46:0/2617354805 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 6 to fsmap v14) v1 ==== 131+0+0 (secure 0 0 0) 0x7f412407a000 con 0x7f413c157990 2023-12-16T21:14:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:added data pool 6 to fsmap 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f411c06ae40 msgr2=0x7f411c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f411c06ae40 0x7f411c06d300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f412c000f60 tx=0x7f412c009040 comp rx=0 tx=0).stop 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 msgr2=0x7f413c1f62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 0x7f413c1f62d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f4124000c00 tx=0x7f41240046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 shutdown_connections 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f411c06ae40 0x7f411c06d300 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f413c157990 0x7f413c1f62d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f413c15e450 0x7f413c1f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 --2- 172.21.15.46:0/2617354805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f413c06aa10 0x7f413c142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.797+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 >> 172.21.15.46:0/2617354805 conn(0x7f413c0b1ad0 msgr2=0x7f413c0b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.798+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 shutdown_connections 2023-12-16T21:14:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.798+0000 7f4141800640 1 -- 172.21.15.46:0/2617354805 wait complete. 2023-12-16T21:14:44.811 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:14:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.923+0000 7f5bc4a3d640 1 Processor -- start 2023-12-16T21:14:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.923+0000 7f5bc4a3d640 1 -- start start 2023-12-16T21:14:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bc4a3d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0160fb0 0x7f5bc0161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bc4a3d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5bc00d0ed0 0x7f5bc00d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bc4a3d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc0161d20 con 0x7f5bc006ea50 2023-12-16T21:14:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bc4a3d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc015e760 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bc4a3d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc015d9a0 con 0x7f5bc0160fb0 2023-12-16T21:14:44.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbe59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5bc006ea50 0x7f5bc006ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60618/0 (socket says 172.21.15.46:60618) 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbe59c640 1 -- 172.21.15.46:0/471293062 learned_addr learned my addr 172.21.15.46:0/471293062 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbed9d640 1 --2- 172.21.15.46:0/471293062 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00d0ed0 0x7f5bc00d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbdd9b640 1 --2- 172.21.15.46:0/471293062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0160fb0 0x7f5bc0161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbd59a640 1 -- 172.21.15.46:0/471293062 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918702248 0 0) 0x7f5bc0161d20 con 0x7f5bc006ea50 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbd59a640 1 -- 172.21.15.46:0/471293062 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bac003520 con 0x7f5bc006ea50 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/471293062 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00d0ed0 msgr2=0x7f5bc00d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbdd9b640 1 --2- 172.21.15.46:0/471293062 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00d0ed0 0x7f5bc00d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/471293062 >> v1:172.21.15.46:6789/0 conn(0x7f5bc006ea50 legacy=0x7f5bc006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/471293062 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc015dc80 con 0x7f5bc0160fb0 2023-12-16T21:14:44.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.924+0000 7f5bbed9d640 1 --2- 172.21.15.46:0/471293062 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00d0ed0 0x7f5bc00d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.925+0000 7f5bbdd9b640 1 --2- 172.21.15.46:0/471293062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0160fb0 0x7f5bc0161440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80097a0 tx=0x7f5ba802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67e706c14140b88 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.925+0000 7f5bbd59a640 1 -- 172.21.15.46:0/471293062 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8031830 con 0x7f5bc0160fb0 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.925+0000 7f5bbd59a640 1 -- 172.21.15.46:0/471293062 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba8031990 con 0x7f5bc0160fb0 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.925+0000 7f5bbd59a640 1 -- 172.21.15.46:0/471293062 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8031d00 con 0x7f5bc0160fb0 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/471293062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0160fb0 msgr2=0x7f5bc0161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/471293062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0160fb0 0x7f5bc0161440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80097a0 tx=0x7f5ba802b8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.928 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/471293062 shutdown_connections 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/471293062 >> v2:172.21.15.121:3300/0 conn(0x7f5bc00d0ed0 0x7f5bc00d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/471293062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc0160fb0 0x7f5bc0161440 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/471293062 >> 172.21.15.46:0/471293062 conn(0x7f5bc006a6b0 msgr2=0x7f5bc006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/471293062 shutdown_connections 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.926+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/471293062 wait complete. 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.927+0000 7f5bc4a3d640 1 Processor -- start 2023-12-16T21:14:44.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.927+0000 7f5bc4a3d640 1 -- start start 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bc4a3d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc006ea50 0x7f5bc015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bc4a3d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 0x7f5bc013ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bc4a3d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc0160fb0 0x7f5bc01e80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bc4a3d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc01e8a10 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bc4a3d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc01e8cf0 con 0x7f5bc0160fb0 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bc4a3d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc015a020 con 0x7f5bc006ea50 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 0x7f5bc013ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:44.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbdd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 0x7f5bc013ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46800/0 (socket says 172.21.15.46:46800) 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/2968508013 learned_addr learned my addr 172.21.15.46:0/2968508013 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbe59c640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc006ea50 0x7f5bc015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbed9d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc0160fb0 0x7f5bc01e80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc006ea50 msgr2=0x7f5bc015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.928+0000 7f5bbdd9b640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc006ea50 0x7f5bc015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc0160fb0 msgr2=0x7f5bc01e80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bbdd9b640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc0160fb0 0x7f5bc01e80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bbdd9b640 1 -- 172.21.15.46:0/2968508013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0009a70 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bbdd9b640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 0x7f5bc013ebf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5ba802b910 tx=0x7f5ba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba803a630 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba803a790 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba8009400 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba803ab40 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.929+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc01e8fd0 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.930+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc015a540 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.930+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5ba803aca0 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.931+0000 7f5bb6ffd640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5ba006ae40 0x7f5ba006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.931+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f5ba80ab2f0 con 0x7f5bc00d0ed0 2023-12-16T21:14:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.931+0000 7f5bbe59c640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5ba006ae40 0x7f5ba006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:44.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.932+0000 7f5bbe59c640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5ba006ae40 0x7f5ba006d300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8005880 tx=0x7f5bb8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:44.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:44.935+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5bc015a540 con 0x7f5bc00d0ed0 2023-12-16T21:14:45.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.146+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bc015a540 con 0x7f5bc00d0ed0 2023-12-16T21:14:45.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.147+0000 7f5bb6ffd640 1 -- 172.21.15.46:0/2968508013 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f5bc015a540 con 0x7f5bc00d0ed0 2023-12-16T21:14:45.149 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:45.149 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:44.132960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:14:45.150 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-12-16T21:14:45.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5ba006ae40 msgr2=0x7f5ba006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5ba006ae40 0x7f5ba006d300 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8005880 tx=0x7f5bb8009040 comp rx=0 tx=0).stop 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 msgr2=0x7f5bc013ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 0x7f5bc013ebf0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5ba802b910 tx=0x7f5ba80046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 shutdown_connections 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5ba006ae40 0x7f5ba006d300 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5bc00d0ed0 0x7f5bc013ebf0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5bc0160fb0 0x7f5bc01e80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 --2- 172.21.15.46:0/2968508013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5bc006ea50 0x7f5bc015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 >> 172.21.15.46:0/2968508013 conn(0x7f5bc006a6b0 msgr2=0x7f5bc00b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 shutdown_connections 2023-12-16T21:14:45.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.149+0000 7f5bc4a3d640 1 -- 172.21.15.46:0/2968508013 wait complete. 2023-12-16T21:14:45.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.267+0000 7fd8b0116640 1 Processor -- start 2023-12-16T21:14:45.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.267+0000 7fd8b0116640 1 -- start start 2023-12-16T21:14:45.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8b0116640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8a815d270 0x7fd8a806ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8b0116640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd8a80709c0 0x7fd8a80cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8b0116640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a80d0c40 con 0x7fd8a80cdd10 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8b0116640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a80ce1a0 con 0x7fd8a815d270 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8b0116640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a80ced90 con 0x7fd8a80709c0 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8adeb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8a815d270 0x7fd8a806ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8adeb2640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd8a815d270 0x7fd8a806ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46810/0 (socket says 172.21.15.46:46810) 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.268+0000 7fd8adeb2640 1 -- 172.21.15.46:0/2124958933 learned_addr learned my addr 172.21.15.46:0/2124958933 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:45.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.269+0000 7fd8ad6b1640 1 --2- 172.21.15.46:0/2124958933 >> v2:172.21.15.121:3300/0 conn(0x7fd8a80709c0 0x7fd8a80cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.269+0000 7fd8adeb2640 1 -- 172.21.15.46:0/2124958933 >> v2:172.21.15.121:3300/0 conn(0x7fd8a80709c0 msgr2=0x7fd8a80cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.269+0000 7fd8adeb2640 1 --2- 172.21.15.46:0/2124958933 >> v2:172.21.15.121:3300/0 conn(0x7fd8a80709c0 0x7fd8a80cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.269+0000 7fd8adeb2640 1 -- 172.21.15.46:0/2124958933 >> v1:172.21.15.121:6789/0 conn(0x7fd8a80cdd10 legacy=0x7fd8a80cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.269+0000 7fd8adeb2640 1 -- 172.21.15.46:0/2124958933 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a80cf070 con 0x7fd8a815d270 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.269+0000 7fd8adeb2640 1 --2- 172.21.15.46:0/2124958933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a815d270 0x7fd8a806ce70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8161fe0 tx=0x7fd8a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38c821fcd36c29d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8aceb0640 1 -- 172.21.15.46:0/2124958933 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a00317e0 con 0x7fd8a815d270 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8aceb0640 1 -- 172.21.15.46:0/2124958933 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0031940 con 0x7fd8a815d270 2023-12-16T21:14:45.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8aceb0640 1 -- 172.21.15.46:0/2124958933 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0031cd0 con 0x7fd8a815d270 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 -- 172.21.15.46:0/2124958933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a815d270 msgr2=0x7fd8a806ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 --2- 172.21.15.46:0/2124958933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a815d270 0x7fd8a806ce70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8161fe0 tx=0x7fd8a002b880 comp rx=0 tx=0).stop 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 -- 172.21.15.46:0/2124958933 shutdown_connections 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 --2- 172.21.15.46:0/2124958933 >> v2:172.21.15.121:3300/0 conn(0x7fd8a80709c0 0x7fd8a80cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 --2- 172.21.15.46:0/2124958933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a815d270 0x7fd8a806ce70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 -- 172.21.15.46:0/2124958933 >> 172.21.15.46:0/2124958933 conn(0x7fd8a80b2c10 msgr2=0x7fd8a80b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 -- 172.21.15.46:0/2124958933 shutdown_connections 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.270+0000 7fd8b0116640 1 -- 172.21.15.46:0/2124958933 wait complete. 2023-12-16T21:14:45.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.271+0000 7fd8b0116640 1 Processor -- start 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.271+0000 7fd8b0116640 1 -- start start 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8b0116640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a80709c0 0x7fd8a815e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8b0116640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 0x7fd8a815eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8b0116640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a815d270 0x7fd8a814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8b0116640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a81601c0 con 0x7fd8a80cdd10 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8b0116640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a815f2e0 con 0x7fd8a815d270 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8b0116640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a815f5b0 con 0x7fd8a80709c0 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8ad6b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 0x7fd8a815eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8ad6b1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 0x7fd8a815eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46816/0 (socket says 172.21.15.46:46816) 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.272+0000 7fd8ad6b1640 1 -- 172.21.15.46:0/4051440924 learned_addr learned my addr 172.21.15.46:0/4051440924 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ad6b1640 1 -- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a80709c0 msgr2=0x7fd8a815e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8adeb2640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a80709c0 0x7fd8a815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ae6b3640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a815d270 0x7fd8a814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ad6b1640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a80709c0 0x7fd8a815e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ad6b1640 1 -- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a815d270 msgr2=0x7fd8a814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ad6b1640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a815d270 0x7fd8a814dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ad6b1640 1 -- 172.21.15.46:0/4051440924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00094d0 con 0x7fd8a80cdd10 2023-12-16T21:14:45.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ae6b3640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a815d270 0x7fd8a814dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8adeb2640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a80709c0 0x7fd8a815e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8ad6b1640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 0x7fd8a815eda0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00c7f0 tx=0x7fd89c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c004220 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c004380 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a815f890 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c007840 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.273+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a815fe50 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.274+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a8067fc0 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.274+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd89c0079a0 con 0x7fd8a80cdd10 2023-12-16T21:14:45.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.275+0000 7fd8a67fc640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd89406ae40 0x7fd89406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.275+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fd89c089410 con 0x7fd8a80cdd10 2023-12-16T21:14:45.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.275+0000 7fd8adeb2640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd89406ae40 0x7fd89406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.275+0000 7fd8adeb2640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd89406ae40 0x7fd89406d300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0036000 tx=0x7fd8a003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:45.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.279+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd89c0543c0 con 0x7fd8a80cdd10 2023-12-16T21:14:45.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.455+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8a8067420 con 0x7fd8a80cdd10 2023-12-16T21:14:45.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.456+0000 7fd8a67fc640 1 -- 172.21.15.46:0/4051440924 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+18361 (secure 0 0 0) 0x7fd89c0579e0 con 0x7fd8a80cdd10 2023-12-16T21:14:45.458 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:45.458 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":36,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:44.132765+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd89406ae40 msgr2=0x7fd89406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd89406ae40 0x7fd89406d300 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0036000 tx=0x7fd8a003d040 comp rx=0 tx=0).stop 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 msgr2=0x7fd8a815eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 0x7fd8a815eda0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd89c00c7f0 tx=0x7fd89c00ccb0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 shutdown_connections 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd89406ae40 0x7fd89406d300 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd8a80cdd10 0x7fd8a815eda0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8a815d270 0x7fd8a814dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 --2- 172.21.15.46:0/4051440924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8a80709c0 0x7fd8a815e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.458+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 >> 172.21.15.46:0/4051440924 conn(0x7fd8a80b2c10 msgr2=0x7fd8a80b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:45.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.459+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 shutdown_connections 2023-12-16T21:14:45.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.459+0000 7fd8b0116640 1 -- 172.21.15.46:0/4051440924 wait complete. 2023-12-16T21:14:45.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:45.470 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2023-12-16T21:14:45.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:45.489 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2023-12-16T21:14:45.548 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2023-12-16T21:14:45.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.701+0000 7f44ea7c5640 1 Processor -- start 2023-12-16T21:14:45.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.701+0000 7f44ea7c5640 1 -- start start 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44ea7c5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44e4146010 0x7f44e40cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44ea7c5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e4146d70 0x7f44e40b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44ea7c5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44e40b4060 0x7f44e40b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44ea7c5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e4161170 con 0x7f44e40b4060 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44ea7c5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e4161450 con 0x7f44e4146010 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44ea7c5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4161730 con 0x7f44e4146d70 2023-12-16T21:14:45.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44e8d62640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44e40b4060 0x7f44e40b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44e8d62640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f44e40b4060 0x7f44e40b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46832/0 (socket says 172.21.15.46:46832) 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44e8d62640 1 -- 172.21.15.46:0/63281591 learned_addr learned my addr 172.21.15.46:0/63281591 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44e37fe640 1 --2- 172.21.15.46:0/63281591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e4146d70 0x7f44e40b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.702+0000 7f44e3fff640 1 --2- 172.21.15.46:0/63281591 >> v2:172.21.15.121:3300/0 conn(0x7f44e4146010 0x7f44e40cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.703+0000 7f44e8d62640 1 -- 172.21.15.46:0/63281591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e4146d70 msgr2=0x7f44e40b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.703+0000 7f44e8d62640 1 --2- 172.21.15.46:0/63281591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e4146d70 0x7f44e40b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.703+0000 7f44e8d62640 1 -- 172.21.15.46:0/63281591 >> v2:172.21.15.121:3300/0 conn(0x7f44e4146010 msgr2=0x7f44e40cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.703+0000 7f44e8d62640 1 --2- 172.21.15.46:0/63281591 >> v2:172.21.15.121:3300/0 conn(0x7f44e4146010 0x7f44e40cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.703+0000 7f44e8d62640 1 -- 172.21.15.46:0/63281591 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4161a10 con 0x7f44e40b4060 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.703+0000 7f44e8d62640 1 --2- 172.21.15.46:0/63281591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e40b4060 0x7f44e40b3230 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f44e40b3ba0 tx=0x7f44dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1028b8ea8265c583 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:45.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44e2ffd640 1 -- 172.21.15.46:0/63281591 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc02ca70 con 0x7f44e40b4060 2023-12-16T21:14:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44e2ffd640 1 -- 172.21.15.46:0/63281591 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44dc02cbd0 con 0x7f44e40b4060 2023-12-16T21:14:45.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44e2ffd640 1 -- 172.21.15.46:0/63281591 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc0343f0 con 0x7f44e40b4060 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 -- 172.21.15.46:0/63281591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e40b4060 msgr2=0x7f44e40b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/63281591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e40b4060 0x7f44e40b3230 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f44e40b3ba0 tx=0x7f44dc02bed0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 -- 172.21.15.46:0/63281591 shutdown_connections 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/63281591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e40b4060 0x7f44e40b3230 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/63281591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e4146d70 0x7f44e40b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/63281591 >> v2:172.21.15.121:3300/0 conn(0x7f44e4146010 0x7f44e40cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 -- 172.21.15.46:0/63281591 >> 172.21.15.46:0/63281591 conn(0x7f44e406a4d0 msgr2=0x7f44e406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 -- 172.21.15.46:0/63281591 shutdown_connections 2023-12-16T21:14:45.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.704+0000 7f44ea7c5640 1 -- 172.21.15.46:0/63281591 wait complete. 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.705+0000 7f44ea7c5640 1 Processor -- start 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 -- start start 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e40b4060 0x7f44e4131b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e4146010 0x7f44e415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 0x7f44e415fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4160230 con 0x7f44e4146d70 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4132130 con 0x7f44e4146010 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44ea7c5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e4132410 con 0x7f44e40b4060 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44e8d62640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 0x7f44e415fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44e8d62640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 0x7f44e415fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46844/0 (socket says 172.21.15.46:46844) 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.706+0000 7f44e8d62640 1 -- 172.21.15.46:0/2482751843 learned_addr learned my addr 172.21.15.46:0/2482751843 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e3fff640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e40b4060 0x7f44e4131b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e37fe640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e4146010 0x7f44e415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e8d62640 1 -- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e40b4060 msgr2=0x7f44e4131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e8d62640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e40b4060 0x7f44e4131b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e8d62640 1 -- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e4146010 msgr2=0x7f44e415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e8d62640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e4146010 0x7f44e415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e8d62640 1 -- 172.21.15.46:0/2482751843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc009b30 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e8d62640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 0x7f44e415fcf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f44dc036040 tx=0x7f44dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc009510 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44dc03e450 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e4146570 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44dc03e800 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.707+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e41329d0 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.708+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e4069400 con 0x7f44e4146d70 2023-12-16T21:14:45.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.709+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f44dc03e960 con 0x7f44e4146d70 2023-12-16T21:14:45.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.709+0000 7f44e0ff9640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44c406ae40 0x7f44c406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:45.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.709+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f44dc0ac420 con 0x7f44e4146d70 2023-12-16T21:14:45.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.709+0000 7f44e3fff640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44c406ae40 0x7f44c406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:45.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.710+0000 7f44e3fff640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44c406ae40 0x7f44c406d300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f44e4134ca0 tx=0x7f44cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:45.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.713+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f44dc077340 con 0x7f44e4146d70 2023-12-16T21:14:45.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.959+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) v1 -- 0x7f44e40676e0 con 0x7f44e4146d70 2023-12-16T21:14:45.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.961+0000 7f44e0ff9640 1 -- 172.21.15.46:0/2482751843 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v16) v1 ==== 210+0+167 (secure 0 0 0) 0x7f44dc07a960 con 0x7f44e4146d70 2023-12-16T21:14:45.962 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:14:45.962 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:14:45.962 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow rw" 2023-12-16T21:14:45.962 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:14:45.962 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2023-12-16T21:14:45.962 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44c406ae40 msgr2=0x7f44c406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44c406ae40 0x7f44c406d300 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f44e4134ca0 tx=0x7f44cc008040 comp rx=0 tx=0).stop 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 msgr2=0x7f44e415fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 0x7f44e415fcf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f44dc036040 tx=0x7f44dc004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 shutdown_connections 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44c406ae40 0x7f44c406d300 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44e4146d70 0x7f44e415fcf0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44e4146010 0x7f44e415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 --2- 172.21.15.46:0/2482751843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44e40b4060 0x7f44e4131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 >> 172.21.15.46:0/2482751843 conn(0x7f44e406a4d0 msgr2=0x7f44e40b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:45.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 shutdown_connections 2023-12-16T21:14:45.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:45.963+0000 7f44ea7c5640 1 -- 172.21.15.46:0/2482751843 wait complete. 2023-12-16T21:14:45.974 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:45.974 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:45.990 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.137+0000 7fd333b78640 1 Processor -- start 2023-12-16T21:14:46.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.137+0000 7fd333b78640 1 -- start start 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd333b78640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd32c15d330 0x7fd32c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd333b78640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd32c0d3900 0x7fd32c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd333b78640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c0b2b10 con 0x7fd32c0d2910 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd333b78640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0d1590 con 0x7fd32c0d3900 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd333b78640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c0d1870 con 0x7fd32c15d330 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd332115640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd32c0d3900 0x7fd32c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd331113640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd32c0d2910 0x7fd32c0d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60634/0 (socket says 172.21.15.46:60634) 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd331113640 1 -- 172.21.15.46:0/2011295691 learned_addr learned my addr 172.21.15.46:0/2011295691 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd331914640 1 --2- 172.21.15.46:0/2011295691 >> v2:172.21.15.121:3300/0 conn(0x7fd32c15d330 0x7fd32c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd332115640 1 -- 172.21.15.46:0/2011295691 >> v2:172.21.15.121:3300/0 conn(0x7fd32c15d330 msgr2=0x7fd32c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:46.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd332115640 1 --2- 172.21.15.46:0/2011295691 >> v2:172.21.15.121:3300/0 conn(0x7fd32c15d330 0x7fd32c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd332115640 1 -- 172.21.15.46:0/2011295691 >> v1:172.21.15.46:6789/0 conn(0x7fd32c0d2910 legacy=0x7fd32c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd332115640 1 -- 172.21.15.46:0/2011295691 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0d1b50 con 0x7fd32c0d3900 2023-12-16T21:14:46.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.138+0000 7fd330912640 1 -- 172.21.15.46:0/2011295691 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876856536 0 0) 0x7fd32c0b2b10 con 0x7fd32c0d2910 2023-12-16T21:14:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.139+0000 7fd332115640 1 --2- 172.21.15.46:0/2011295691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c0d3900 0x7fd32c0b6390 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0d0d10 tx=0x7fd32402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c36abe6a1e0732 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.139+0000 7fd330912640 1 -- 172.21.15.46:0/2011295691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32402cbf0 con 0x7fd32c0d3900 2023-12-16T21:14:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.139+0000 7fd330912640 1 -- 172.21.15.46:0/2011295691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32402cd50 con 0x7fd32c0d3900 2023-12-16T21:14:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.139+0000 7fd330912640 1 -- 172.21.15.46:0/2011295691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd324034660 con 0x7fd32c0d3900 2023-12-16T21:14:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 -- 172.21.15.46:0/2011295691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c0d3900 msgr2=0x7fd32c0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 --2- 172.21.15.46:0/2011295691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c0d3900 0x7fd32c0b6390 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0d0d10 tx=0x7fd32402be40 comp rx=0 tx=0).stop 2023-12-16T21:14:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 -- 172.21.15.46:0/2011295691 shutdown_connections 2023-12-16T21:14:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 --2- 172.21.15.46:0/2011295691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c0d3900 0x7fd32c0b6390 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 --2- 172.21.15.46:0/2011295691 >> v2:172.21.15.121:3300/0 conn(0x7fd32c15d330 0x7fd32c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 -- 172.21.15.46:0/2011295691 >> 172.21.15.46:0/2011295691 conn(0x7fd32c072020 msgr2=0x7fd32c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 -- 172.21.15.46:0/2011295691 shutdown_connections 2023-12-16T21:14:46.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.140+0000 7fd333b78640 1 -- 172.21.15.46:0/2011295691 wait complete. 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.141+0000 7fd333b78640 1 Processor -- start 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.141+0000 7fd333b78640 1 -- start start 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd333b78640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd32c0d2910 0x7fd32c1534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd333b78640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd32c0d3900 0x7fd32c153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd333b78640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 0x7fd32c1f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd333b78640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c156870 con 0x7fd32c15d330 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd333b78640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0c9180 con 0x7fd32c0d2910 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd333b78640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c155c60 con 0x7fd32c0d3900 2023-12-16T21:14:46.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 0x7fd32c1f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 0x7fd32c1f6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46870/0 (socket says 172.21.15.46:46870) 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 -- 172.21.15.46:0/201659579 learned_addr learned my addr 172.21.15.46:0/201659579 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd331914640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd32c0d2910 0x7fd32c1534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd331113640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd32c0d3900 0x7fd32c153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 -- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd32c0d3900 msgr2=0x7fd32c153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd32c0d3900 0x7fd32c153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 -- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd32c0d2910 msgr2=0x7fd32c1534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd32c0d2910 0x7fd32c1534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 -- 172.21.15.46:0/201659579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd324009a70 con 0x7fd32c15d330 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.142+0000 7fd332115640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 0x7fd32c1f6670 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd324009e80 tx=0x7fd324004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.143+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd324040650 con 0x7fd32c15d330 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.143+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3240407d0 con 0x7fd32c15d330 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.143+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c155f40 con 0x7fd32c15d330 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.143+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd324040b80 con 0x7fd32c15d330 2023-12-16T21:14:46.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.143+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c156500 con 0x7fd32c15d330 2023-12-16T21:14:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.144+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c069400 con 0x7fd32c15d330 2023-12-16T21:14:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.144+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd324040ce0 con 0x7fd32c15d330 2023-12-16T21:14:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.144+0000 7fd3227fc640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd31006ae70 0x7fd31006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.144+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fd3240ac400 con 0x7fd32c15d330 2023-12-16T21:14:46.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.145+0000 7fd331914640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd31006ae70 0x7fd31006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.145+0000 7fd331914640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd31006ae70 0x7fd31006d330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd328006fd0 tx=0x7fd328008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.149+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd324077580 con 0x7fd32c15d330 2023-12-16T21:14:46.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.327+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd32c0d1060 con 0x7fd32c15d330 2023-12-16T21:14:46.329 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.329+0000 7fd3227fc640 1 -- 172.21.15.46:0/201659579 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7fd32407aba0 con 0x7fd32c15d330 2023-12-16T21:14:46.329 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:46.330 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.330+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd31006ae70 msgr2=0x7fd31006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.330+0000 7fd333b78640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd31006ae70 0x7fd31006d330 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd328006fd0 tx=0x7fd328008040 comp rx=0 tx=0).stop 2023-12-16T21:14:46.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.330+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 msgr2=0x7fd32c1f6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.330+0000 7fd333b78640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 0x7fd32c1f6670 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd324009e80 tx=0x7fd324004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 shutdown_connections 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd31006ae70 0x7fd31006d330 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd32c15d330 0x7fd32c1f6670 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd32c0d3900 0x7fd32c153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 --2- 172.21.15.46:0/201659579 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd32c0d2910 0x7fd32c1534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 >> 172.21.15.46:0/201659579 conn(0x7fd32c072020 msgr2=0x7fd32c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 shutdown_connections 2023-12-16T21:14:46.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.331+0000 7fd333b78640 1 -- 172.21.15.46:0/201659579 wait complete. 2023-12-16T21:14:46.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:46.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:46.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.451+0000 7fcd08689640 1 Processor -- start 2023-12-16T21:14:46.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.451+0000 7fcd08689640 1 -- start start 2023-12-16T21:14:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd08689640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd0015d330 0x7fcd000b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd08689640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d3900 0x7fcd000b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd08689640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd000b2b10 con 0x7fcd000d2910 2023-12-16T21:14:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd08689640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd000d1590 con 0x7fcd0015d330 2023-12-16T21:14:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd08689640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000d1870 con 0x7fcd000d3900 2023-12-16T21:14:46.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd0015d330 0x7fcd000b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcd0015d330 0x7fcd000b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46884/0 (socket says 172.21.15.46:46884) 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd05c24640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcd000d2910 0x7fcd000d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60636/0 (socket says 172.21.15.46:60636) 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 -- 172.21.15.46:0/2246689410 learned_addr learned my addr 172.21.15.46:0/2246689410 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06c26640 1 --2- 172.21.15.46:0/2246689410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d3900 0x7fcd000b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 -- 172.21.15.46:0/2246689410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d3900 msgr2=0x7fcd000b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 --2- 172.21.15.46:0/2246689410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d3900 0x7fcd000b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 -- 172.21.15.46:0/2246689410 >> v1:172.21.15.46:6789/0 conn(0x7fcd000d2910 legacy=0x7fcd000d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd05423640 1 -- 172.21.15.46:0/2246689410 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181832080 0 0) 0x7fcd000b2b10 con 0x7fcd000d2910 2023-12-16T21:14:46.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.452+0000 7fcd06425640 1 -- 172.21.15.46:0/2246689410 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd000d1b50 con 0x7fcd0015d330 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.453+0000 7fcd06425640 1 --2- 172.21.15.46:0/2246689410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8b80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009870 tx=0x7fccfc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b238f356a8fe5e81 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.453+0000 7fcd05423640 1 -- 172.21.15.46:0/2246689410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc031640 con 0x7fcd0015d330 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.453+0000 7fcd05423640 1 -- 172.21.15.46:0/2246689410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc0317a0 con 0x7fcd0015d330 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.453+0000 7fcd05423640 1 -- 172.21.15.46:0/2246689410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc031b30 con 0x7fcd0015d330 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 -- 172.21.15.46:0/2246689410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 msgr2=0x7fcd000b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 --2- 172.21.15.46:0/2246689410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8b80 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009870 tx=0x7fccfc02b8f0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 -- 172.21.15.46:0/2246689410 shutdown_connections 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 --2- 172.21.15.46:0/2246689410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d3900 0x7fcd000b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 --2- 172.21.15.46:0/2246689410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8b80 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 -- 172.21.15.46:0/2246689410 >> 172.21.15.46:0/2246689410 conn(0x7fcd00072020 msgr2=0x7fcd00072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:46.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 -- 172.21.15.46:0/2246689410 shutdown_connections 2023-12-16T21:14:46.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.454+0000 7fcd08689640 1 -- 172.21.15.46:0/2246689410 wait complete. 2023-12-16T21:14:46.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.455+0000 7fcd08689640 1 Processor -- start 2023-12-16T21:14:46.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.455+0000 7fcd08689640 1 -- start start 2023-12-16T21:14:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd08689640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d2910 0x7fcd0014b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd08689640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd000d3900 0x7fcd000b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd08689640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.459 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd08689640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015f260 con 0x7fcd0015d330 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd08689640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015e650 con 0x7fcd000d3900 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd08689640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015e930 con 0x7fcd000d2910 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46898/0 (socket says 172.21.15.46:46898) 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 -- 172.21.15.46:0/4255426052 learned_addr learned my addr 172.21.15.46:0/4255426052 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06425640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d2910 0x7fcd0014b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd05c24640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd000d3900 0x7fcd000b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 -- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d2910 msgr2=0x7fcd0014b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d2910 0x7fcd0014b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 -- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd000d3900 msgr2=0x7fcd000b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd000d3900 0x7fcd000b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 -- 172.21.15.46:0/4255426052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc0094d0 con 0x7fcd0015d330 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.456+0000 7fcd06c26640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8bf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fccf8007e40 tx=0x7fccf800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.457+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf8014620 con 0x7fcd0015d330 2023-12-16T21:14:46.460 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.457+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf800ee90 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.457+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0015ec10 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.457+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf80149e0 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.457+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0015d920 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.458+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0015f410 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.458+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fccf8014b40 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.458+0000 7fccf6ffd640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcce406ae70 0x7fcce406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.459+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fccf808a190 con 0x7fcd0015d330 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.459+0000 7fcd06425640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcce406ae70 0x7fcce406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.461 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.459+0000 7fcd06425640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcce406ae70 0x7fcce406d330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fccfc036000 tx=0x7fccfc03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.463 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.463+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcd0015f410 con 0x7fcd0015d330 2023-12-16T21:14:46.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.638+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcd000d2dc0 con 0x7fcd0015d330 2023-12-16T21:14:46.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.639+0000 7fccf6ffd640 1 -- 172.21.15.46:0/4255426052 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7fccf8054f80 con 0x7fcd0015d330 2023-12-16T21:14:46.639 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcce406ae70 msgr2=0x7fcce406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcce406ae70 0x7fcce406d330 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fccfc036000 tx=0x7fccfc03c040 comp rx=0 tx=0).stop 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 msgr2=0x7fcd000b8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8bf0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fccf8007e40 tx=0x7fccf800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 shutdown_connections 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcce406ae70 0x7fcce406d330 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd0015d330 0x7fcd000b8bf0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd000d3900 0x7fcd000b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 --2- 172.21.15.46:0/4255426052 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd000d2910 0x7fcd0014b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 >> 172.21.15.46:0/4255426052 conn(0x7fcd00072020 msgr2=0x7fcd00072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 shutdown_connections 2023-12-16T21:14:46.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.641+0000 7fcd08689640 1 -- 172.21.15.46:0/4255426052 wait complete. 2023-12-16T21:14:46.652 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:46.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.761+0000 7fc221fa5640 1 Processor -- start 2023-12-16T21:14:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.761+0000 7fc221fa5640 1 -- start start 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc221fa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 0x7fc21c161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc221fa5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc21c0d0ed0 0x7fc21c0d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc221fa5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21c161d20 con 0x7fc21c06ea50 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc221fa5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21c15e760 con 0x7fc21c0d0ed0 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc221fa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c15d9a0 con 0x7fc21c160fb0 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21b7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc21c06ea50 0x7fc21c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60648/0 (socket says 172.21.15.46:60648) 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21b7fe640 1 -- 172.21.15.46:0/1877999479 learned_addr learned my addr 172.21.15.46:0/1877999479 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21affd640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 0x7fc21c161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721720620 0 0) 0x7fc21c161d20 con 0x7fc21c06ea50 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21bfff640 1 --2- 172.21.15.46:0/1877999479 >> v2:172.21.15.121:3300/0 conn(0x7fc21c0d0ed0 0x7fc21c0d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204003520 con 0x7fc21c06ea50 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593730384 0 0) 0x7fc204003520 con 0x7fc21c06ea50 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 msgr2=0x7fc21c161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:46.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 0x7fc21c161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 >> v2:172.21.15.121:3300/0 conn(0x7fc21c0d0ed0 msgr2=0x7fc21c0d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 --2- 172.21.15.46:0/1877999479 >> v2:172.21.15.121:3300/0 conn(0x7fc21c0d0ed0 0x7fc21c0d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c15dc80 con 0x7fc21c06ea50 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fc20c001670 con 0x7fc21c06ea50 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 >> v1:172.21.15.46:6789/0 conn(0x7fc21c06ea50 legacy=0x7fc21c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 0x7fc204003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc204003ef0 0x7fc2040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc204004d00 0x7fc204005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c15e760 con 0x7fc204003ef0 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c15d9a0 con 0x7fc204004d00 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.762+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20400b980 con 0x7fc21c160fb0 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21affd640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc204003ef0 0x7fc2040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21b7fe640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 0x7fc204003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21bfff640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc204004d00 0x7fc204005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21affd640 1 -- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 msgr2=0x7fc204003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:46.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21affd640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc21c160fb0 0x7fc204003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21affd640 1 -- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc204004d00 msgr2=0x7fc204005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:46.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21affd640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc204004d00 0x7fc204005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.763+0000 7fc21affd640 1 -- 172.21.15.46:0/1877999479 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc204003520 con 0x7fc204003ef0 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc21affd640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc204003ef0 0x7fc2040047c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc210009ac0 tx=0x7fc2040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dc8b1a60f28521 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21002c6b0 con 0x7fc204003ef0 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21002c810 con 0x7fc204003ef0 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc21a7fc640 1 -- 172.21.15.46:0/1877999479 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21002cba0 con 0x7fc204003ef0 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc221fa5640 1 -- 172.21.15.46:0/1877999479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc204003ef0 msgr2=0x7fc2040047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1877999479 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc204003ef0 0x7fc2040047c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc210009ac0 tx=0x7fc2040057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc21bfff640 1 -- 172.21.15.46:0/1877999479 reap_dead start 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc221fa5640 1 -- 172.21.15.46:0/1877999479 shutdown_connections 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc221fa5640 1 -- 172.21.15.46:0/1877999479 >> 172.21.15.46:0/1877999479 conn(0x7fc21c06a6b0 msgr2=0x7fc21c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc221fa5640 1 -- 172.21.15.46:0/1877999479 shutdown_connections 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.764+0000 7fc221fa5640 1 -- 172.21.15.46:0/1877999479 wait complete. 2023-12-16T21:14:46.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 Processor -- start 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 -- start start 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc204003ef0 0x7fc21c1f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 0x7fc21c15a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc21c15fb90 0x7fc21c160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c160550 con 0x7fc21c159890 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c1f6ea0 con 0x7fc21c15fb90 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.766+0000 7fc221fa5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc21c1344e0 con 0x7fc204003ef0 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 0x7fc21c15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 0x7fc21c15a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46918/0 (socket says 172.21.15.46:46918) 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 -- 172.21.15.46:0/1200341236 learned_addr learned my addr 172.21.15.46:0/1200341236 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21b7fe640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc204003ef0 0x7fc21c1f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21bfff640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc21c15fb90 0x7fc21c160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 -- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc204003ef0 msgr2=0x7fc21c1f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc204003ef0 0x7fc21c1f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 -- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc21c15fb90 msgr2=0x7fc21c160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc21c15fb90 0x7fc21c160010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 -- 172.21.15.46:0/1200341236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21c1347c0 con 0x7fc21c159890 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc21affd640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 0x7fc21c15a520 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc210009ac0 tx=0x7fc210004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210003810 con 0x7fc21c159890 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc210003970 con 0x7fc21c159890 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210003d00 con 0x7fc21c159890 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21c134aa0 con 0x7fc21c159890 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.767+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc21c0cb7f0 con 0x7fc21c159890 2023-12-16T21:14:46.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.768+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21c15b290 con 0x7fc21c159890 2023-12-16T21:14:46.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.769+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc210040450 con 0x7fc21c159890 2023-12-16T21:14:46.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.769+0000 7fc203fff640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc20806ae70 0x7fc20806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:46.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.769+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fc2100ab500 con 0x7fc21c159890 2023-12-16T21:14:46.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.769+0000 7fc21b7fe640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc20806ae70 0x7fc20806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:46.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.770+0000 7fc21b7fe640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc20806ae70 0x7fc20806d330 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc20c007fd0 tx=0x7fc20c0078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:46.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.773+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc210076560 con 0x7fc21c159890 2023-12-16T21:14:46.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.948+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc21c069400 con 0x7fc21c159890 2023-12-16T21:14:46.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.948+0000 7fc203fff640 1 -- 172.21.15.46:0/1200341236 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc210003e60 con 0x7fc21c159890 2023-12-16T21:14:46.949 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc20806ae70 msgr2=0x7fc20806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc20806ae70 0x7fc20806d330 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc20c007fd0 tx=0x7fc20c0078f0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 msgr2=0x7fc21c15a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 0x7fc21c15a520 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc210009ac0 tx=0x7fc210004730 comp rx=0 tx=0).stop 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 shutdown_connections 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc20806ae70 0x7fc20806d330 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc21c159890 0x7fc21c15a520 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc21c15fb90 0x7fc21c160010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 --2- 172.21.15.46:0/1200341236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc204003ef0 0x7fc21c1f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 >> 172.21.15.46:0/1200341236 conn(0x7fc21c06a6b0 msgr2=0x7fc21c140850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 shutdown_connections 2023-12-16T21:14:46.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:46.950+0000 7fc221fa5640 1 -- 172.21.15.46:0/1200341236 wait complete. 2023-12-16T21:14:46.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.070+0000 7f0fb3f3e640 1 Processor -- start 2023-12-16T21:14:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.070+0000 7f0fb3f3e640 1 -- start start 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb3f3e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fac15d2b0 0x7f0fac071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb3f3e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0fac0ae480 0x7f0fac0721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb3f3e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fac0ae910 con 0x7f0fac072840 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb3f3e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fac0b2200 con 0x7f0fac15d2b0 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb3f3e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fac0b24e0 con 0x7f0fac0ae480 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb1cda640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0fac15d2b0 0x7f0fac071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb24db640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0fac072840 0x7f0fac06d340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60656/0 (socket says 172.21.15.46:60656) 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb24db640 1 -- 172.21.15.46:0/1492554930 learned_addr learned my addr 172.21.15.46:0/1492554930 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb14d9640 1 --2- 172.21.15.46:0/1492554930 >> v2:172.21.15.121:3300/0 conn(0x7f0fac0ae480 0x7f0fac0721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb1cda640 1 -- 172.21.15.46:0/1492554930 >> v2:172.21.15.121:3300/0 conn(0x7f0fac0ae480 msgr2=0x7f0fac0721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb1cda640 1 --2- 172.21.15.46:0/1492554930 >> v2:172.21.15.121:3300/0 conn(0x7f0fac0ae480 0x7f0fac0721c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb1cda640 1 -- 172.21.15.46:0/1492554930 >> v1:172.21.15.46:6789/0 conn(0x7f0fac072840 legacy=0x7f0fac06d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.071+0000 7f0fb1cda640 1 -- 172.21.15.46:0/1492554930 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac0b40a0 con 0x7f0fac15d2b0 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.072+0000 7f0fb1cda640 1 --2- 172.21.15.46:0/1492554930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac15d2b0 0x7f0fac071c80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0fac160e00 tx=0x7f0fa802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5abce6bf64b01587 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.072+0000 7f0fb0cd8640 1 -- 172.21.15.46:0/1492554930 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa802e070 con 0x7f0fac15d2b0 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.072+0000 7f0fb0cd8640 1 -- 172.21.15.46:0/1492554930 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa80315a0 con 0x7f0fac15d2b0 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.072+0000 7f0fb0cd8640 1 -- 172.21.15.46:0/1492554930 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8031930 con 0x7f0fac15d2b0 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/1492554930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac15d2b0 msgr2=0x7f0fac071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/1492554930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac15d2b0 0x7f0fac071c80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0fac160e00 tx=0x7f0fa802b840 comp rx=0 tx=0).stop 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/1492554930 shutdown_connections 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/1492554930 >> v2:172.21.15.121:3300/0 conn(0x7f0fac0ae480 0x7f0fac0721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/1492554930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac15d2b0 0x7f0fac071c80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/1492554930 >> 172.21.15.46:0/1492554930 conn(0x7f0fac0b4720 msgr2=0x7f0fac0b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/1492554930 shutdown_connections 2023-12-16T21:14:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.073+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/1492554930 wait complete. 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.074+0000 7f0fb3f3e640 1 Processor -- start 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.074+0000 7f0fb3f3e640 1 -- start start 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb3f3e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fac072840 0x7f0fac151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb3f3e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0fac15d2b0 0x7f0fac150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb3f3e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 0x7f0fac158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb3f3e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac0b1e80 con 0x7f0fac0b19d0 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb3f3e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac0b1760 con 0x7f0fac072840 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb3f3e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac1516e0 con 0x7f0fac15d2b0 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 0x7f0fac158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 0x7f0fac158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46932/0 (socket says 172.21.15.46:46932) 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 -- 172.21.15.46:0/473266834 learned_addr learned my addr 172.21.15.46:0/473266834 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb1cda640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fac072840 0x7f0fac151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb14d9640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0fac15d2b0 0x7f0fac150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 -- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0fac15d2b0 msgr2=0x7f0fac150ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0fac15d2b0 0x7f0fac150ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 -- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fac072840 msgr2=0x7f0fac151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fac072840 0x7f0fac151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 -- 172.21.15.46:0/473266834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa8009400 con 0x7f0fac0b19d0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb14d9640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0fac15d2b0 0x7f0fac150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb1cda640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fac072840 0x7f0fac151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fb24db640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 0x7f0fac158030 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400cbc0 tx=0x7f0fa400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.075+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400ad40 con 0x7f0fac0b19d0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.076+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa4012400 con 0x7f0fac0b19d0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.076+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac0b2af0 con 0x7f0fac0b19d0 2023-12-16T21:14:47.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.076+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa400aea0 con 0x7f0fac0b19d0 2023-12-16T21:14:47.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.076+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fac151ca0 con 0x7f0fac0b19d0 2023-12-16T21:14:47.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.077+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fac0b2950 con 0x7f0fac0b19d0 2023-12-16T21:14:47.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.077+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0fa4007070 con 0x7f0fac0b19d0 2023-12-16T21:14:47.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.077+0000 7f0fa27fc640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0f9006ae40 0x7f0f9006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.077+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f0fa40895b0 con 0x7f0fac0b19d0 2023-12-16T21:14:47.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.078+0000 7f0fb1cda640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0f9006ae40 0x7f0f9006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.078+0000 7f0fb1cda640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0f9006ae40 0x7f0f9006d300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8036000 tx=0x7f0fa803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.081+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0fac0b2950 con 0x7f0fac0b19d0 2023-12-16T21:14:47.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.255+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0fac0b2950 con 0x7f0fac0b19d0 2023-12-16T21:14:47.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.256+0000 7f0fa27fc640 1 -- 172.21.15.46:0/473266834 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0fac0b2950 con 0x7f0fac0b19d0 2023-12-16T21:14:47.257 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0f9006ae40 msgr2=0x7f0f9006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0f9006ae40 0x7f0f9006d300 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8036000 tx=0x7f0fa803d040 comp rx=0 tx=0).stop 2023-12-16T21:14:47.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 msgr2=0x7f0fac158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 0x7f0fac158030 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0fa400cbc0 tx=0x7f0fa400a430 comp rx=0 tx=0).stop 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 shutdown_connections 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0f9006ae40 0x7f0f9006d300 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fac0b19d0 0x7f0fac158030 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0fac15d2b0 0x7f0fac150ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 --2- 172.21.15.46:0/473266834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0fac072840 0x7f0fac151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 >> 172.21.15.46:0/473266834 conn(0x7f0fac0b4720 msgr2=0x7f0fac06c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 shutdown_connections 2023-12-16T21:14:47.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.258+0000 7f0fb3f3e640 1 -- 172.21.15.46:0/473266834 wait complete. 2023-12-16T21:14:47.269 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:14:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.378+0000 7f5239f53640 1 Processor -- start 2023-12-16T21:14:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.378+0000 7f5239f53640 1 -- start start 2023-12-16T21:14:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.379+0000 7f5239f53640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52340cf320 0x7f52340d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.379+0000 7f5239f53640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 0x7f52340cd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.379+0000 7f5239f53640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52340b3f10 con 0x7f523415b130 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.379+0000 7f5239f53640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523415fd20 con 0x7f523415f760 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.379+0000 7f5239f53640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52340d0730 con 0x7f52340cf320 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.379+0000 7f5233fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 0x7f52340cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5232ffd640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52340cf320 0x7f52340d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5233fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 0x7f52340cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:53316/0 (socket says 172.21.15.46:53316) 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f52337fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f523415b130 0x7f52340cec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:50650/0 (socket says 172.21.15.46:50650) 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5233fff640 1 -- 172.21.15.46:0/4006953156 learned_addr learned my addr 172.21.15.46:0/4006953156 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5232ffd640 1 -- 172.21.15.46:0/4006953156 >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 msgr2=0x7f52340cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5232ffd640 1 --2- 172.21.15.46:0/4006953156 >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 0x7f52340cd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5232ffd640 1 -- 172.21.15.46:0/4006953156 >> v1:172.21.15.121:6789/0 conn(0x7f523415b130 legacy=0x7f52340cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5232ffd640 1 -- 172.21.15.46:0/4006953156 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52340d0a50 con 0x7f52340cf320 2023-12-16T21:14:47.381 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.380+0000 7f5233fff640 1 --2- 172.21.15.46:0/4006953156 >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 0x7f52340cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f5232ffd640 1 --2- 172.21.15.46:0/4006953156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52340cf320 0x7f52340d01f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5228009d90 tx=0x7f522802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30be1aeadebabe3a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f52327fc640 1 -- 172.21.15.46:0/4006953156 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522802cc50 con 0x7f52340cf320 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f52327fc640 1 -- 172.21.15.46:0/4006953156 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522802cdb0 con 0x7f52340cf320 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f52327fc640 1 -- 172.21.15.46:0/4006953156 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228034660 con 0x7f52340cf320 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f5239f53640 1 -- 172.21.15.46:0/4006953156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52340cf320 msgr2=0x7f52340d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f5239f53640 1 --2- 172.21.15.46:0/4006953156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52340cf320 0x7f52340d01f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5228009d90 tx=0x7f522802bea0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f5239f53640 1 -- 172.21.15.46:0/4006953156 shutdown_connections 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.381+0000 7f5239f53640 1 --2- 172.21.15.46:0/4006953156 >> v2:172.21.15.121:3300/0 conn(0x7f523415f760 0x7f52340cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.382+0000 7f5239f53640 1 --2- 172.21.15.46:0/4006953156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52340cf320 0x7f52340d01f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.382+0000 7f5239f53640 1 -- 172.21.15.46:0/4006953156 >> 172.21.15.46:0/4006953156 conn(0x7f52340b2c10 msgr2=0x7f52340b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.382+0000 7f5239f53640 1 -- 172.21.15.46:0/4006953156 shutdown_connections 2023-12-16T21:14:47.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.382+0000 7f5239f53640 1 -- 172.21.15.46:0/4006953156 wait complete. 2023-12-16T21:14:47.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.383+0000 7f5239f53640 1 Processor -- start 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.383+0000 7f5239f53640 1 -- start start 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5239f53640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52340cf320 0x7f52340cae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5239f53640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f523415b130 0x7f52341560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5239f53640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 0x7f52341f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5239f53640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523415c140 con 0x7f523415f760 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5239f53640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523406c9a0 con 0x7f52340cf320 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5239f53640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523415b810 con 0x7f523415b130 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 0x7f52341f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 0x7f52341f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46934/0 (socket says 172.21.15.46:46934) 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 -- 172.21.15.46:0/2465583312 learned_addr learned my addr 172.21.15.46:0/2465583312 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5232ffd640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f523415b130 0x7f52341560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f52337fe640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52340cf320 0x7f52340cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 -- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f523415b130 msgr2=0x7f52341560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f523415b130 0x7f52341560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 -- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52340cf320 msgr2=0x7f52340cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52340cf320 0x7f52340cae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 -- 172.21.15.46:0/2465583312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c0094d0 con 0x7f523415f760 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.384+0000 7f5233fff640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 0x7f52341f6560 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f522c00e890 tx=0x7f522c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.385+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c004020 con 0x7f523415f760 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.385+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f522c004180 con 0x7f523415f760 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.385+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522c00c940 con 0x7f523415f760 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.385+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5228009a70 con 0x7f523415f760 2023-12-16T21:14:47.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.385+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523415bdd0 con 0x7f523415f760 2023-12-16T21:14:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.386+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5234067fc0 con 0x7f523415f760 2023-12-16T21:14:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.386+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f522c00caa0 con 0x7f523415f760 2023-12-16T21:14:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.386+0000 7f521bfff640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f522006ae40 0x7f522006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.387+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f522c089710 con 0x7f523415f760 2023-12-16T21:14:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.387+0000 7f52337fe640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f522006ae40 0x7f522006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.387+0000 7f52337fe640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f522006ae40 0x7f522006d300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5224000f60 tx=0x7f5224000ed0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.391+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f522c0545c0 con 0x7f523415f760 2023-12-16T21:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.565+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f5234067420 con 0x7f523415f760 2023-12-16T21:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.566+0000 7f521bfff640 1 -- 172.21.15.46:0/2465583312 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7f522c057be0 con 0x7f523415f760 2023-12-16T21:14:47.567 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f522006ae40 msgr2=0x7f522006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f522006ae40 0x7f522006d300 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5224000f60 tx=0x7f5224000ed0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 msgr2=0x7f52341f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 0x7f52341f6560 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f522c00e890 tx=0x7f522c00ed50 comp rx=0 tx=0).stop 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 shutdown_connections 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f522006ae40 0x7f522006d300 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f523415f760 0x7f52341f6560 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f523415b130 0x7f52341560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 --2- 172.21.15.46:0/2465583312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52340cf320 0x7f52340cae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.567+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 >> 172.21.15.46:0/2465583312 conn(0x7f52340b2c10 msgr2=0x7f52340b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.568+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 shutdown_connections 2023-12-16T21:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.568+0000 7f5239f53640 1 -- 172.21.15.46:0/2465583312 wait complete. 2023-12-16T21:14:47.579 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.687+0000 7fdbc2f01640 1 Processor -- start 2023-12-16T21:14:47.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.687+0000 7fdbc2f01640 1 -- start start 2023-12-16T21:14:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc2f01640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdbbc144980 0x7fdbbc15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc2f01640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc2f01640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc071770 con 0x7fdbbc0b4370 2023-12-16T21:14:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc2f01640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0ccdc0 con 0x7fdbbc144980 2023-12-16T21:14:47.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc2f01640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0cd0a0 con 0x7fdbbc0ceb70 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdbbc144980 0x7fdbbc15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdbbc144980 0x7fdbbc15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46940/0 (socket says 172.21.15.46:46940) 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/37663137 learned_addr learned my addr 172.21.15.46:0/37663137 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbbbfff640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/37663137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 msgr2=0x7fdbbc0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/37663137 >> v1:172.21.15.121:6789/0 conn(0x7fdbbc0b4370 legacy=0x7fdbbc071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/37663137 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0cd380 con 0x7fdbbc144980 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.688+0000 7fdbbbfff640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.689+0000 7fdbc0c9d640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc144980 0x7fdbbc15c7c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009870 tx=0x7fdbb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c716c79834c48eb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.689+0000 7fdbbb7fe640 1 -- 172.21.15.46:0/37663137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0030840 con 0x7fdbbc144980 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.689+0000 7fdbbb7fe640 1 -- 172.21.15.46:0/37663137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb00309a0 con 0x7fdbbc144980 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.689+0000 7fdbbb7fe640 1 -- 172.21.15.46:0/37663137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0030d30 con 0x7fdbbc144980 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 -- 172.21.15.46:0/37663137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc144980 msgr2=0x7fdbbc15c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc144980 0x7fdbbc15c7c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009870 tx=0x7fdbb002b980 comp rx=0 tx=0).stop 2023-12-16T21:14:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 -- 172.21.15.46:0/37663137 shutdown_connections 2023-12-16T21:14:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/37663137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc144980 0x7fdbbc15c7c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 -- 172.21.15.46:0/37663137 >> 172.21.15.46:0/37663137 conn(0x7fdbbc06a4d0 msgr2=0x7fdbbc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 -- 172.21.15.46:0/37663137 shutdown_connections 2023-12-16T21:14:47.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.690+0000 7fdbc2f01640 1 -- 172.21.15.46:0/37663137 wait complete. 2023-12-16T21:14:47.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.691+0000 7fdbc2f01640 1 Processor -- start 2023-12-16T21:14:47.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.691+0000 7fdbc2f01640 1 -- start start 2023-12-16T21:14:47.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc2f01640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 0x7fdbbc142840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc2f01640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0ac670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc2f01640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbbc144980 0x7fdbbc0aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc2f01640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0af7a0 con 0x7fdbbc0b4370 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc2f01640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0b1890 con 0x7fdbbc144980 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc2f01640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0aee70 con 0x7fdbbc0ceb70 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 0x7fdbbc142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 0x7fdbbc142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46942/0 (socket says 172.21.15.46:46942) 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/2650271298 learned_addr learned my addr 172.21.15.46:0/2650271298 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbbbfff640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc149e640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbbc144980 0x7fdbbc0aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 msgr2=0x7fdbbc0ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0ac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbbc144980 msgr2=0x7fdbbc0aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbbc144980 0x7fdbbc0aa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 -- 172.21.15.46:0/2650271298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00094d0 con 0x7fdbbc0b4370 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbbbfff640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:47.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.692+0000 7fdbc0c9d640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 0x7fdbbc142840 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.693+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0039a90 con 0x7fdbbc0b4370 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.693+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb0039bf0 con 0x7fdbbc0b4370 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.693+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0043410 con 0x7fdbbc0b4370 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.693+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc0af150 con 0x7fdbbc0b4370 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.693+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc142d80 con 0x7fdbbc0b4370 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.694+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc069400 con 0x7fdbbc0b4370 2023-12-16T21:14:47.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.694+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdbb002f390 con 0x7fdbbc0b4370 2023-12-16T21:14:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.694+0000 7fdbb97fa640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdba006ae40 0x7fdba006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.694+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fdbb00ab920 con 0x7fdbbc0b4370 2023-12-16T21:14:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.695+0000 7fdbbbfff640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdba006ae40 0x7fdba006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:47.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.695+0000 7fdbbbfff640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdba006ae40 0x7fdba006d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0b0aa0 tx=0x7fdba8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:47.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.699+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdbb00768d0 con 0x7fdbbc0b4370 2023-12-16T21:14:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.889+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdbbc06ab00 con 0x7fdbbc0b4370 2023-12-16T21:14:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.889+0000 7fdbb97fa640 1 -- 172.21.15.46:0/2650271298 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdbb0034080 con 0x7fdbbc0b4370 2023-12-16T21:14:47.890 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdba006ae40 msgr2=0x7fdba006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdba006ae40 0x7fdba006d300 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc0b0aa0 tx=0x7fdba8008040 comp rx=0 tx=0).stop 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 msgr2=0x7fdbbc142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 0x7fdbbc142840 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0000c00 tx=0x7fdbb0004730 comp rx=0 tx=0).stop 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 shutdown_connections 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdba006ae40 0x7fdba006d300 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdbbc144980 0x7fdbbc0aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdbbc0ceb70 0x7fdbbc0ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 --2- 172.21.15.46:0/2650271298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbbc0b4370 0x7fdbbc142840 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:47.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 >> 172.21.15.46:0/2650271298 conn(0x7fdbbc06a4d0 msgr2=0x7fdbbc143be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:47.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 shutdown_connections 2023-12-16T21:14:47.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:47.891+0000 7fdbc2f01640 1 -- 172.21.15.46:0/2650271298 wait complete. 2023-12-16T21:14:47.902 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:47.902 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:47.918 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:48.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.064+0000 7f4b0cc89640 1 Processor -- start 2023-12-16T21:14:48.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.064+0000 7f4b0cc89640 1 -- start start 2023-12-16T21:14:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.065+0000 7f4b0cc89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b080d2910 0x7f4b080d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.065+0000 7f4b0cc89640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b080d3730 0x7f4b080b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.065+0000 7f4b0cc89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b080b2b10 con 0x7f4b0815d330 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.065+0000 7f4b0cc89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b080d3bc0 con 0x7f4b080d2910 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.065+0000 7f4b0cc89640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b080d1590 con 0x7f4b080d3730 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.065+0000 7f4b05d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b080d2910 0x7f4b080d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b05d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4b080d2910 0x7f4b080d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46950/0 (socket says 172.21.15.46:46950) 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b05d9b640 1 -- 172.21.15.46:0/319273662 learned_addr learned my addr 172.21.15.46:0/319273662 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b06d9d640 1 --2- 172.21.15.46:0/319273662 >> v2:172.21.15.121:3300/0 conn(0x7f4b080d3730 0x7f4b080b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b05d9b640 1 -- 172.21.15.46:0/319273662 >> v2:172.21.15.121:3300/0 conn(0x7f4b080d3730 msgr2=0x7f4b080b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b05d9b640 1 --2- 172.21.15.46:0/319273662 >> v2:172.21.15.121:3300/0 conn(0x7f4b080d3730 0x7f4b080b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b05d9b640 1 -- 172.21.15.46:0/319273662 >> v1:172.21.15.121:6789/0 conn(0x7f4b0815d330 legacy=0x7f4b080b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:48.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b05d9b640 1 -- 172.21.15.46:0/319273662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b080d1870 con 0x7f4b080d2910 2023-12-16T21:14:48.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.066+0000 7f4b06d9d640 1 --2- 172.21.15.46:0/319273662 >> v2:172.21.15.121:3300/0 conn(0x7f4b080d3730 0x7f4b080b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b05d9b640 1 --2- 172.21.15.46:0/319273662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d2910 0x7f4b080d31f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4b0806c500 tx=0x7f4af002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed065a344402744b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0559a640 1 -- 172.21.15.46:0/319273662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0031870 con 0x7f4b080d2910 2023-12-16T21:14:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0559a640 1 -- 172.21.15.46:0/319273662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af00319d0 con 0x7f4b080d2910 2023-12-16T21:14:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0559a640 1 -- 172.21.15.46:0/319273662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af0031d60 con 0x7f4b080d2910 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 -- 172.21.15.46:0/319273662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d2910 msgr2=0x7f4b080d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/319273662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d2910 0x7f4b080d31f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4b0806c500 tx=0x7f4af002b910 comp rx=0 tx=0).stop 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 -- 172.21.15.46:0/319273662 shutdown_connections 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/319273662 >> v2:172.21.15.121:3300/0 conn(0x7f4b080d3730 0x7f4b080b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/319273662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d2910 0x7f4b080d31f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 -- 172.21.15.46:0/319273662 >> 172.21.15.46:0/319273662 conn(0x7f4b08072020 msgr2=0x7f4b08072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 -- 172.21.15.46:0/319273662 shutdown_connections 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.067+0000 7f4b0cc89640 1 -- 172.21.15.46:0/319273662 wait complete. 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.069+0000 7f4b0cc89640 1 Processor -- start 2023-12-16T21:14:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.069+0000 7f4b0cc89640 1 -- start start 2023-12-16T21:14:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b080d2910 0x7f4b080cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 0x7f4b080c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b0815d330 0x7f4b08153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080c6cd0 con 0x7f4b080d3730 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080cc380 con 0x7f4b080d2910 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b080cc660 con 0x7f4b0815d330 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 0x7f4b080c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 0x7f4b080c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46960/0 (socket says 172.21.15.46:46960) 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 -- 172.21.15.46:0/1477135712 learned_addr learned my addr 172.21.15.46:0/1477135712 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0659c640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b080d2910 0x7f4b080cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b06d9d640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b0815d330 0x7f4b08153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 -- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b0815d330 msgr2=0x7f4b08153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b0815d330 0x7f4b08153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 -- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b080d2910 msgr2=0x7f4b080cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b080d2910 0x7f4b080cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 -- 172.21.15.46:0/1477135712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00094d0 con 0x7f4b080d3730 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0659c640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b080d2910 0x7f4b080cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b05d9b640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 0x7f4b080c8180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4af002b950 tx=0x7f4af00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af003ad30 con 0x7f4b080d3730 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af00443f0 con 0x7f4b080d3730 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b080cc940 con 0x7f4b080d3730 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.070+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af00447a0 con 0x7f4b080d3730 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.071+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b080c86c0 con 0x7f4b080d3730 2023-12-16T21:14:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.072+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b080c6e40 con 0x7f4b080d3730 2023-12-16T21:14:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.072+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4af004d450 con 0x7f4b080d3730 2023-12-16T21:14:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.072+0000 7f4afeffd640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4ae806ae70 0x7f4ae806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.072+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f4af002d040 con 0x7f4b080d3730 2023-12-16T21:14:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.072+0000 7f4b0659c640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4ae806ae70 0x7f4ae806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.073+0000 7f4b0659c640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4ae806ae70 0x7f4ae806d330 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b00007fd0 tx=0x7f4b00009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.076+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4b080c6e40 con 0x7f4b080d3730 2023-12-16T21:14:48.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.252+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b080c6e10 con 0x7f4b080d3730 2023-12-16T21:14:48.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.253+0000 7f4afeffd640 1 -- 172.21.15.46:0/1477135712 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7f4af0035020 con 0x7f4b080d3730 2023-12-16T21:14:48.254 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:48.254 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4ae806ae70 msgr2=0x7f4ae806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4ae806ae70 0x7f4ae806d330 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b00007fd0 tx=0x7f4b00009040 comp rx=0 tx=0).stop 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 msgr2=0x7f4b080c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 0x7f4b080c8180 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4af002b950 tx=0x7f4af00046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 shutdown_connections 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4ae806ae70 0x7f4ae806d330 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b080d3730 0x7f4b080c8180 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b0815d330 0x7f4b08153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 --2- 172.21.15.46:0/1477135712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b080d2910 0x7f4b080cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 >> 172.21.15.46:0/1477135712 conn(0x7f4b08072020 msgr2=0x7f4b080b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:48.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 shutdown_connections 2023-12-16T21:14:48.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.255+0000 7f4b0cc89640 1 -- 172.21.15.46:0/1477135712 wait complete. 2023-12-16T21:14:48.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:48.267 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:48.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.376+0000 7f047411d640 1 Processor -- start 2023-12-16T21:14:48.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.376+0000 7f047411d640 1 -- start start 2023-12-16T21:14:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f047411d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c06eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f047411d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f046c140f10 0x7f046c13f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f047411d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c15ba40 con 0x7f046c06f040 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f047411d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046c15c6d0 con 0x7f046c140f10 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f047411d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c13e530 con 0x7f046c15c2f0 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f046c140f10 0x7f046c13f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f0471eb9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c06eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f046c140f10 0x7f046c13f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:53356/0 (socket says 172.21.15.46:53356) 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 -- 172.21.15.46:0/1518150772 learned_addr learned my addr 172.21.15.46:0/1518150772 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04716b8640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f046c06f040 0x7f046c140800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:50690/0 (socket says 172.21.15.46:50690) 2023-12-16T21:14:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 -- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 msgr2=0x7f046c06eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 --2- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c06eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 -- 172.21.15.46:0/1518150772 >> v1:172.21.15.121:6789/0 conn(0x7f046c06f040 legacy=0x7f046c140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f04726ba640 1 -- 172.21.15.46:0/1518150772 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c06f4a0 con 0x7f046c140f10 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.377+0000 7f0471eb9640 1 --2- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c06eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.378+0000 7f04726ba640 1 --2- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c140f10 0x7f046c13f330 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0464009eb0 tx=0x7f046402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1182301eb2fac server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.378+0000 7f0470eb7640 1 -- 172.21.15.46:0/1518150772 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046402cad0 con 0x7f046c140f10 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.378+0000 7f0470eb7640 1 -- 172.21.15.46:0/1518150772 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046402cc30 con 0x7f046c140f10 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.378+0000 7f0470eb7640 1 -- 172.21.15.46:0/1518150772 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04640343f0 con 0x7f046c140f10 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 -- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c140f10 msgr2=0x7f046c13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 --2- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c140f10 0x7f046c13f330 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0464009eb0 tx=0x7f046402bf30 comp rx=0 tx=0).stop 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 -- 172.21.15.46:0/1518150772 shutdown_connections 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 --2- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c140f10 0x7f046c13f330 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 --2- 172.21.15.46:0/1518150772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c06eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 -- 172.21.15.46:0/1518150772 >> 172.21.15.46:0/1518150772 conn(0x7f046c06a6b0 msgr2=0x7f046c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 -- 172.21.15.46:0/1518150772 shutdown_connections 2023-12-16T21:14:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.379+0000 7f047411d640 1 -- 172.21.15.46:0/1518150772 wait complete. 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.380+0000 7f047411d640 1 Processor -- start 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.380+0000 7f047411d640 1 -- start start 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f047411d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c06f040 0x7f046c1549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f047411d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 0x7f046c13ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f047411d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f047411d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c139cf0 con 0x7f046c140f10 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f047411d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c153a00 con 0x7f046c06f040 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f047411d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c139f40 con 0x7f046c15c2f0 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 0x7f046c13ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 0x7f046c13ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46970/0 (socket says 172.21.15.46:46970) 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 -- 172.21.15.46:0/1945705850 learned_addr learned my addr 172.21.15.46:0/1945705850 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:48.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f0471eb9640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c06f040 0x7f046c1549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04726ba640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 -- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 msgr2=0x7f046c151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 -- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c06f040 msgr2=0x7f046c1549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.381+0000 7f04716b8640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c06f040 0x7f046c1549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f04716b8640 1 -- 172.21.15.46:0/1945705850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0468009400 con 0x7f046c140f10 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f04716b8640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 0x7f046c13ade0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f045c00c4d0 tx=0x7f045c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c007480 con 0x7f046c140f10 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c0075e0 con 0x7f046c140f10 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0464009b30 con 0x7f046c140f10 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c0049e0 con 0x7f046c140f10 2023-12-16T21:14:48.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.382+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c13a500 con 0x7f046c140f10 2023-12-16T21:14:48.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.383+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c069400 con 0x7f046c140f10 2023-12-16T21:14:48.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.383+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f045c004b40 con 0x7f046c140f10 2023-12-16T21:14:48.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.384+0000 7f04627fc640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f045006ae40 0x7f045006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.384+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f045c088c60 con 0x7f046c140f10 2023-12-16T21:14:48.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.384+0000 7f0471eb9640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f045006ae40 0x7f045006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.384+0000 7f0471eb9640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f045006ae40 0x7f045006d300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0468009c30 tx=0x7f0468009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.388+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f045c053c30 con 0x7f046c140f10 2023-12-16T21:14:48.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.563+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f046c070d30 con 0x7f046c140f10 2023-12-16T21:14:48.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.563+0000 7f04627fc640 1 -- 172.21.15.46:0/1945705850 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f045c0180a0 con 0x7f046c140f10 2023-12-16T21:14:48.564 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f045006ae40 msgr2=0x7f045006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f045006ae40 0x7f045006d300 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0468009c30 tx=0x7f0468009190 comp rx=0 tx=0).stop 2023-12-16T21:14:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 msgr2=0x7f046c13ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 0x7f046c13ade0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f045c00c4d0 tx=0x7f045c00c990 comp rx=0 tx=0).stop 2023-12-16T21:14:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 shutdown_connections 2023-12-16T21:14:48.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f045006ae40 0x7f045006d300 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f046c140f10 0x7f046c13ade0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f046c15c2f0 0x7f046c151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 --2- 172.21.15.46:0/1945705850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f046c06f040 0x7f046c1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 >> 172.21.15.46:0/1945705850 conn(0x7f046c06a6b0 msgr2=0x7f046c15dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 shutdown_connections 2023-12-16T21:14:48.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.565+0000 7f047411d640 1 -- 172.21.15.46:0/1945705850 wait complete. 2023-12-16T21:14:48.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:48.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.687+0000 7f96d2105640 1 Processor -- start 2023-12-16T21:14:48.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.687+0000 7f96d2105640 1 -- start start 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.687+0000 7f96d2105640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f96cc155790 0x7f96cc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.687+0000 7f96d2105640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96cc15b5e0 0x7f96cc15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.687+0000 7f96d2105640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc155b70 con 0x7f96cc06a4d0 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96d2105640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc06ee30 con 0x7f96cc15b5e0 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96d2105640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc06f3b0 con 0x7f96cc155790 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96cc15b5e0 0x7f96cc15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f96cc15b5e0 0x7f96cc15ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46976/0 (socket says 172.21.15.46:46976) 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96cbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f96cc06a4d0 0x7f96cc06a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60670/0 (socket says 172.21.15.46:60670) 2023-12-16T21:14:48.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 -- 172.21.15.46:0/1138028393 learned_addr learned my addr 172.21.15.46:0/1138028393 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:48.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96cb7fe640 1 --2- 172.21.15.46:0/1138028393 >> v2:172.21.15.121:3300/0 conn(0x7f96cc155790 0x7f96cc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 -- 172.21.15.46:0/1138028393 >> v2:172.21.15.121:3300/0 conn(0x7f96cc155790 msgr2=0x7f96cc06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:48.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 --2- 172.21.15.46:0/1138028393 >> v2:172.21.15.121:3300/0 conn(0x7f96cc155790 0x7f96cc06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 -- 172.21.15.46:0/1138028393 >> v1:172.21.15.46:6789/0 conn(0x7f96cc06a4d0 legacy=0x7f96cc06a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96caffd640 1 -- 172.21.15.46:0/1138028393 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc15c170 con 0x7f96cc15b5e0 2023-12-16T21:14:48.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.688+0000 7f96ca7fc640 1 -- 172.21.15.46:0/1138028393 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028065739 0 0) 0x7f96cc155b70 con 0x7f96cc06a4d0 2023-12-16T21:14:48.691 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96caffd640 1 --2- 172.21.15.46:0/1138028393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc15b5e0 0x7f96cc15ba90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f96cc072720 tx=0x7f96b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ad89e7bdbd53f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96ca7fc640 1 -- 172.21.15.46:0/1138028393 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4030860 con 0x7f96cc15b5e0 2023-12-16T21:14:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96ca7fc640 1 -- 172.21.15.46:0/1138028393 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b40309c0 con 0x7f96cc15b5e0 2023-12-16T21:14:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96ca7fc640 1 -- 172.21.15.46:0/1138028393 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b4030d50 con 0x7f96cc15b5e0 2023-12-16T21:14:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96d2105640 1 -- 172.21.15.46:0/1138028393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc15b5e0 msgr2=0x7f96cc15ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96d2105640 1 --2- 172.21.15.46:0/1138028393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc15b5e0 0x7f96cc15ba90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f96cc072720 tx=0x7f96b402b9a0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96d2105640 1 -- 172.21.15.46:0/1138028393 shutdown_connections 2023-12-16T21:14:48.693 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96d2105640 1 --2- 172.21.15.46:0/1138028393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc15b5e0 0x7f96cc15ba90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96d2105640 1 --2- 172.21.15.46:0/1138028393 >> v2:172.21.15.121:3300/0 conn(0x7f96cc155790 0x7f96cc06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.689+0000 7f96d2105640 1 -- 172.21.15.46:0/1138028393 >> 172.21.15.46:0/1138028393 conn(0x7f96cc0b1e50 msgr2=0x7f96cc0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:48.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.690+0000 7f96d2105640 1 -- 172.21.15.46:0/1138028393 shutdown_connections 2023-12-16T21:14:48.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.690+0000 7f96d2105640 1 -- 172.21.15.46:0/1138028393 wait complete. 2023-12-16T21:14:48.694 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.691+0000 7f96d2105640 1 Processor -- start 2023-12-16T21:14:48.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.691+0000 7f96d2105640 1 -- start start 2023-12-16T21:14:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 0x7f96cc1398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96cc155790 0x7f96cc139030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96cc15b5e0 0x7f96cc15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc139e10 con 0x7f96cc06a4d0 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc14da80 con 0x7f96cc155790 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc150460 con 0x7f96cc15b5e0 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 0x7f96cc1398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 0x7f96cc1398d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46984/0 (socket says 172.21.15.46:46984) 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 -- 172.21.15.46:0/2680494147 learned_addr learned my addr 172.21.15.46:0/2680494147 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96caffd640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96cc155790 0x7f96cc139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 -- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96cc15b5e0 msgr2=0x7f96cc15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cbfff640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96cc15b5e0 0x7f96cc15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96cc15b5e0 0x7f96cc15a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 -- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96cc155790 msgr2=0x7f96cc139030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96cc155790 0x7f96cc139030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 -- 172.21.15.46:0/2680494147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b40094d0 con 0x7f96cc06a4d0 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96caffd640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96cc155790 0x7f96cc139030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:48.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96cb7fe640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 0x7f96cc1398d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00e9b0 tx=0x7f96bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc009670 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96bc00c9a0 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00cd50 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc150740 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.692+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc14d0d0 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.693+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc069400 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.694+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f96bc013470 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.694+0000 7f96c3fff640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96ac06ae40 0x7f96ac06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.694+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f96bc089ea0 con 0x7f96cc06a4d0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.694+0000 7f96caffd640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96ac06ae40 0x7f96ac06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:48.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.695+0000 7f96caffd640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96ac06ae40 0x7f96ac06d300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f96b4036000 tx=0x7f96b403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:48.700 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.698+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f96bc054c90 con 0x7f96cc06a4d0 2023-12-16T21:14:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.875+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96cc159c50 con 0x7f96cc06a4d0 2023-12-16T21:14:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.876+0000 7f96c3fff640 1 -- 172.21.15.46:0/2680494147 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96bc021020 con 0x7f96cc06a4d0 2023-12-16T21:14:48.877 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96ac06ae40 msgr2=0x7f96ac06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96ac06ae40 0x7f96ac06d300 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f96b4036000 tx=0x7f96b403b040 comp rx=0 tx=0).stop 2023-12-16T21:14:48.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 msgr2=0x7f96cc1398d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 0x7f96cc1398d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00e9b0 tx=0x7f96bc00ee70 comp rx=0 tx=0).stop 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 shutdown_connections 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f96ac06ae40 0x7f96ac06d300 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96cc15b5e0 0x7f96cc15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96cc155790 0x7f96cc139030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 --2- 172.21.15.46:0/2680494147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96cc06a4d0 0x7f96cc1398d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 >> 172.21.15.46:0/2680494147 conn(0x7f96cc0b1e50 msgr2=0x7f96cc0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 shutdown_connections 2023-12-16T21:14:48.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.878+0000 7f96d2105640 1 -- 172.21.15.46:0/2680494147 wait complete. 2023-12-16T21:14:48.889 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:48.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.998+0000 7f58ad483640 1 Processor -- start 2023-12-16T21:14:48.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.998+0000 7f58ad483640 1 -- start start 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58ad483640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a80b1400 0x7f58a815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58ad483640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58a815e730 0x7f58a815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58ad483640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a815e2a0 con 0x7f58a8143490 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58ad483640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a815b520 con 0x7f58a815e730 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58ad483640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a8143870 con 0x7f58a80b1400 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58a815e730 0x7f58a815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a6ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f58a8143490 0x7f58a80b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60672/0 (socket says 172.21.15.46:60672) 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a6ffd640 1 -- 172.21.15.46:0/3675372800 learned_addr learned my addr 172.21.15.46:0/3675372800 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:49.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a77fe640 1 -- 172.21.15.46:0/3675372800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a80b1400 msgr2=0x7f58a815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a77fe640 1 --2- 172.21.15.46:0/3675372800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a80b1400 0x7f58a815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a77fe640 1 -- 172.21.15.46:0/3675372800 >> v1:172.21.15.46:6789/0 conn(0x7f58a8143490 legacy=0x7f58a80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a77fe640 1 -- 172.21.15.46:0/3675372800 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a815b240 con 0x7f58a815e730 2023-12-16T21:14:49.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:48.999+0000 7f58a5ffb640 1 -- 172.21.15.46:0/3675372800 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829500537 0 0) 0x7f58a815e2a0 con 0x7f58a8143490 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.000+0000 7f58a77fe640 1 --2- 172.21.15.46:0/3675372800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a815aa20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ace4cce30f0f0e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.000+0000 7f58a5ffb640 1 -- 172.21.15.46:0/3675372800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0031830 con 0x7f58a815e730 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.000+0000 7f58a5ffb640 1 -- 172.21.15.46:0/3675372800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a0031990 con 0x7f58a815e730 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.000+0000 7f58a5ffb640 1 -- 172.21.15.46:0/3675372800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0031d20 con 0x7f58a815e730 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 -- 172.21.15.46:0/3675372800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 msgr2=0x7f58a815aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 --2- 172.21.15.46:0/3675372800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a815aa20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 -- 172.21.15.46:0/3675372800 shutdown_connections 2023-12-16T21:14:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 --2- 172.21.15.46:0/3675372800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a815aa20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 --2- 172.21.15.46:0/3675372800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a80b1400 0x7f58a815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 -- 172.21.15.46:0/3675372800 >> 172.21.15.46:0/3675372800 conn(0x7f58a806a4d0 msgr2=0x7f58a806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 -- 172.21.15.46:0/3675372800 shutdown_connections 2023-12-16T21:14:49.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.001+0000 7f58ad483640 1 -- 172.21.15.46:0/3675372800 wait complete. 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.002+0000 7f58ad483640 1 Processor -- start 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.002+0000 7f58ad483640 1 -- start start 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58ad483640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58a80b1400 0x7f58a8159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58ad483640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a8143490 0x7f58a81419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58ad483640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a8141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58ad483640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81e5880 con 0x7f58a815e730 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58ad483640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81e4ac0 con 0x7f58a80b1400 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58ad483640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a81e4da0 con 0x7f58a8143490 2023-12-16T21:14:49.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a8141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a8141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47006/0 (socket says 172.21.15.46:47006) 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 -- 172.21.15.46:0/3668847248 learned_addr learned my addr 172.21.15.46:0/3668847248 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a67fc640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a8143490 0x7f58a81419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 -- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a8143490 msgr2=0x7f58a81419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58a80b1400 0x7f58a8159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a8143490 0x7f58a81419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 -- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58a80b1400 msgr2=0x7f58a8159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58a80b1400 0x7f58a8159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 -- 172.21.15.46:0/3668847248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0009400 con 0x7f58a815e730 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.003+0000 7f58a77fe640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a8141f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.004+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a003a9e0 con 0x7f58a815e730 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.004+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a003ab40 con 0x7f58a815e730 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.004+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58a81e5080 con 0x7f58a815e730 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.004+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a00443f0 con 0x7f58a815e730 2023-12-16T21:14:49.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.004+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a812f8a0 con 0x7f58a815e730 2023-12-16T21:14:49.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.005+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a8069400 con 0x7f58a815e730 2023-12-16T21:14:49.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.005+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f58a004d450 con 0x7f58a815e730 2023-12-16T21:14:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.005+0000 7f589f7fe640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f588806ae70 0x7f588806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.006+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f58a00ab990 con 0x7f58a815e730 2023-12-16T21:14:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.006+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f588806ae70 0x7f588806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.006+0000 7f58a6ffd640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f588806ae70 0x7f588806d330 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58980012e0 tx=0x7f5898009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.010 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.010+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f58a0076820 con 0x7f58a815e730 2023-12-16T21:14:49.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.184+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58a80ae380 con 0x7f58a815e730 2023-12-16T21:14:49.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.184+0000 7f589f7fe640 1 -- 172.21.15.46:0/3668847248 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58a0035070 con 0x7f58a815e730 2023-12-16T21:14:49.185 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f588806ae70 msgr2=0x7f588806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f588806ae70 0x7f588806d330 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58980012e0 tx=0x7f5898009040 comp rx=0 tx=0).stop 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 msgr2=0x7f58a8141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a8141f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a00046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 shutdown_connections 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f588806ae70 0x7f588806d330 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58a815e730 0x7f58a8141f20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58a8143490 0x7f58a81419e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 --2- 172.21.15.46:0/3668847248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58a80b1400 0x7f58a8159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 >> 172.21.15.46:0/3668847248 conn(0x7f58a806a4d0 msgr2=0x7f58a80ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 shutdown_connections 2023-12-16T21:14:49.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.186+0000 7f58ad483640 1 -- 172.21.15.46:0/3668847248 wait complete. 2023-12-16T21:14:49.197 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:14:49.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.306+0000 7fb5540e0640 1 Processor -- start 2023-12-16T21:14:49.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.306+0000 7fb5540e0640 1 -- start start 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb5540e0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb54c1415f0 0x7fb54c141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb5540e0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb54c06bbf0 0x7fb54c0b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb5540e0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c0b2eb0 con 0x7fb54c144a00 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb5540e0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c0b3190 con 0x7fb54c1415f0 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb5540e0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c0b3470 con 0x7fb54c06bbf0 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb54c1415f0 0x7fb54c141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb54c1415f0 0x7fb54c141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47016/0 (socket says 172.21.15.46:47016) 2023-12-16T21:14:49.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 -- 172.21.15.46:0/1763317564 learned_addr learned my addr 172.21.15.46:0/1763317564 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb55267d640 1 --2- 172.21.15.46:0/1763317564 >> v2:172.21.15.121:3300/0 conn(0x7fb54c06bbf0 0x7fb54c0b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 -- 172.21.15.46:0/1763317564 >> v2:172.21.15.121:3300/0 conn(0x7fb54c06bbf0 msgr2=0x7fb54c0b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 --2- 172.21.15.46:0/1763317564 >> v2:172.21.15.121:3300/0 conn(0x7fb54c06bbf0 0x7fb54c0b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 -- 172.21.15.46:0/1763317564 >> v1:172.21.15.121:6789/0 conn(0x7fb54c144a00 legacy=0x7fb54c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:49.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.307+0000 7fb551e7c640 1 -- 172.21.15.46:0/1763317564 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0b3750 con 0x7fb54c1415f0 2023-12-16T21:14:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.308+0000 7fb551e7c640 1 --2- 172.21.15.46:0/1763317564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c1415f0 0x7fb54c141e00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb54c066aa0 tx=0x7fb54402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2a8354057ea2e2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.308+0000 7fb550e7a640 1 -- 172.21.15.46:0/1763317564 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544031870 con 0x7fb54c1415f0 2023-12-16T21:14:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.308+0000 7fb550e7a640 1 -- 172.21.15.46:0/1763317564 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5440319d0 con 0x7fb54c1415f0 2023-12-16T21:14:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.308+0000 7fb550e7a640 1 -- 172.21.15.46:0/1763317564 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb544031d60 con 0x7fb54c1415f0 2023-12-16T21:14:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 -- 172.21.15.46:0/1763317564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c1415f0 msgr2=0x7fb54c141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 --2- 172.21.15.46:0/1763317564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c1415f0 0x7fb54c141e00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb54c066aa0 tx=0x7fb54402b910 comp rx=0 tx=0).stop 2023-12-16T21:14:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 -- 172.21.15.46:0/1763317564 shutdown_connections 2023-12-16T21:14:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 --2- 172.21.15.46:0/1763317564 >> v2:172.21.15.121:3300/0 conn(0x7fb54c06bbf0 0x7fb54c0b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 --2- 172.21.15.46:0/1763317564 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c1415f0 0x7fb54c141e00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 -- 172.21.15.46:0/1763317564 >> 172.21.15.46:0/1763317564 conn(0x7fb54c070e70 msgr2=0x7fb54c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 -- 172.21.15.46:0/1763317564 shutdown_connections 2023-12-16T21:14:49.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.309+0000 7fb5540e0640 1 -- 172.21.15.46:0/1763317564 wait complete. 2023-12-16T21:14:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.310+0000 7fb5540e0640 1 Processor -- start 2023-12-16T21:14:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.310+0000 7fb5540e0640 1 -- start start 2023-12-16T21:14:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb5540e0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb54c06bbf0 0x7fb54c155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb5540e0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb54c1415f0 0x7fb54c1577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb5540e0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 0x7fb54c1fa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb5540e0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c153da0 con 0x7fb54c144a00 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb5540e0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c157e50 con 0x7fb54c06bbf0 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb5540e0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c155c70 con 0x7fb54c1415f0 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 0x7fb54c1fa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 0x7fb54c1fa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47018/0 (socket says 172.21.15.46:47018) 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 -- 172.21.15.46:0/645006408 learned_addr learned my addr 172.21.15.46:0/645006408 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb551e7c640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb54c06bbf0 0x7fb54c155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55167b640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb54c1415f0 0x7fb54c1577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 -- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb54c1415f0 msgr2=0x7fb54c1577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb54c1415f0 0x7fb54c1577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 -- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb54c06bbf0 msgr2=0x7fb54c155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb54c06bbf0 0x7fb54c155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb55267d640 1 -- 172.21.15.46:0/645006408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440094d0 con 0x7fb54c144a00 2023-12-16T21:14:49.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.311+0000 7fb551e7c640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb54c06bbf0 0x7fb54c155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.312+0000 7fb55267d640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 0x7fb54c1fa2e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb53c007e40 tx=0x7fb53c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.312+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c014620 con 0x7fb54c144a00 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.312+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53c00ee90 con 0x7fb54c144a00 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.312+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c155f50 con 0x7fb54c144a00 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.312+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c0149e0 con 0x7fb54c144a00 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.312+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c156510 con 0x7fb54c144a00 2023-12-16T21:14:49.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.313+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c069400 con 0x7fb54c144a00 2023-12-16T21:14:49.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.313+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb53c014b40 con 0x7fb54c144a00 2023-12-16T21:14:49.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.314+0000 7fb54a7fc640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.314+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fb53c08a0d0 con 0x7fb54c144a00 2023-12-16T21:14:49.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.314+0000 7fb551e7c640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.314+0000 7fb551e7c640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb544036000 tx=0x7fb54403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.318+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb53c055080 con 0x7fb54c144a00 2023-12-16T21:14:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.494+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7fb54c06f7f0 con 0x7fb54c144a00 2023-12-16T21:14:49.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.494+0000 7fb54a7fc640 1 -- 172.21.15.46:0/645006408 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7fb53c0586a0 con 0x7fb54c144a00 2023-12-16T21:14:49.495 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 msgr2=0x7fb53806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb544036000 tx=0x7fb54403c040 comp rx=0 tx=0).stop 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 msgr2=0x7fb54c1fa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 0x7fb54c1fa2e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb53c007e40 tx=0x7fb53c00e4b0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 shutdown_connections 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb53806ae40 0x7fb53806d300 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb54c144a00 0x7fb54c1fa2e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb54c1415f0 0x7fb54c1577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 --2- 172.21.15.46:0/645006408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb54c06bbf0 0x7fb54c155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 >> 172.21.15.46:0/645006408 conn(0x7fb54c070e70 msgr2=0x7fb54c0af750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 shutdown_connections 2023-12-16T21:14:49.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.496+0000 7fb5540e0640 1 -- 172.21.15.46:0/645006408 wait complete. 2023-12-16T21:14:49.507 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:49.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.614+0000 7f8975161640 1 Processor -- start 2023-12-16T21:14:49.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.614+0000 7f8975161640 1 -- start start 2023-12-16T21:14:49.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f8975161640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89700ae480 0x7f897006d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f8975161640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8970072390 0x7f897006d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f8975161640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89700ae910 con 0x7f897015d2b0 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f8975161640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89700b2200 con 0x7f89700ae480 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f8975161640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89700b24e0 con 0x7f8970072390 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f896ed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f897015d2b0 0x7f8970071c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:60676/0 (socket says 172.21.15.46:60676) 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f896ed9d640 1 -- 172.21.15.46:0/3857092413 learned_addr learned my addr 172.21.15.46:0/3857092413 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f896f59e640 1 --2- 172.21.15.46:0/3857092413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8970072390 0x7f897006d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f896e59c640 1 --2- 172.21.15.46:0/3857092413 >> v2:172.21.15.121:3300/0 conn(0x7f89700ae480 0x7f897006d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f896dd9b640 1 -- 172.21.15.46:0/3857092413 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261666040 0 0) 0x7f89700ae910 con 0x7f897015d2b0 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.615+0000 7f896dd9b640 1 -- 172.21.15.46:0/3857092413 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f895c003520 con 0x7f897015d2b0 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.616+0000 7f896f59e640 1 -- 172.21.15.46:0/3857092413 >> v2:172.21.15.121:3300/0 conn(0x7f89700ae480 msgr2=0x7f897006d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.616+0000 7f896f59e640 1 --2- 172.21.15.46:0/3857092413 >> v2:172.21.15.121:3300/0 conn(0x7f89700ae480 0x7f897006d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.616+0000 7f896f59e640 1 -- 172.21.15.46:0/3857092413 >> v1:172.21.15.46:6789/0 conn(0x7f897015d2b0 legacy=0x7f8970071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.616+0000 7f896f59e640 1 -- 172.21.15.46:0/3857092413 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89700b40a0 con 0x7f8970072390 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.616+0000 7f896f59e640 1 --2- 172.21.15.46:0/3857092413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8970072390 0x7f897006d880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89680097a0 tx=0x7f896802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4427a053fab2e3f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f896dd9b640 1 -- 172.21.15.46:0/3857092413 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896802e070 con 0x7f8970072390 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f896dd9b640 1 -- 172.21.15.46:0/3857092413 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89680315a0 con 0x7f8970072390 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f896dd9b640 1 -- 172.21.15.46:0/3857092413 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8968031910 con 0x7f8970072390 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 -- 172.21.15.46:0/3857092413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8970072390 msgr2=0x7f897006d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 --2- 172.21.15.46:0/3857092413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8970072390 0x7f897006d880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89680097a0 tx=0x7f896802b840 comp rx=0 tx=0).stop 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 -- 172.21.15.46:0/3857092413 shutdown_connections 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 --2- 172.21.15.46:0/3857092413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8970072390 0x7f897006d880 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 --2- 172.21.15.46:0/3857092413 >> v2:172.21.15.121:3300/0 conn(0x7f89700ae480 0x7f897006d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 -- 172.21.15.46:0/3857092413 >> 172.21.15.46:0/3857092413 conn(0x7f89700b4720 msgr2=0x7f89700b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:49.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 -- 172.21.15.46:0/3857092413 shutdown_connections 2023-12-16T21:14:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.617+0000 7f8975161640 1 -- 172.21.15.46:0/3857092413 wait complete. 2023-12-16T21:14:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 Processor -- start 2023-12-16T21:14:49.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 -- start start 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 0x7f89700ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f897015d2b0 0x7f89700ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8970158cd0 0x7f8970145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897015b990 con 0x7f8970072390 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897009a430 con 0x7f8970158cd0 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f8975161640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8970159190 con 0x7f897015d2b0 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.619+0000 7f896ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 0x7f89700ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 0x7f89700ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47034/0 (socket says 172.21.15.46:47034) 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 -- 172.21.15.46:0/272308088 learned_addr learned my addr 172.21.15.46:0/272308088 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896f59e640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8970158cd0 0x7f8970145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896e59c640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f897015d2b0 0x7f89700ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 -- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f897015d2b0 msgr2=0x7f89700ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f897015d2b0 0x7f89700ac720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 -- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8970158cd0 msgr2=0x7f8970145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8970158cd0 0x7f8970145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 -- 172.21.15.46:0/272308088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8968009400 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896f59e640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8970158cd0 0x7f8970145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f896ed9d640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 0x7f89700ac1e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f896000caa0 tx=0x7f896000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896000ac00 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f896000ad60 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8960012620 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89700b2af0 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.620+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8970159750 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.621+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8970069400 con 0x7f8970072390 2023-12-16T21:14:49.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.622+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8960007070 con 0x7f8970072390 2023-12-16T21:14:49.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.622+0000 7f89677fe640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f895006ae40 0x7f895006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:49.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.622+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8960089720 con 0x7f8970072390 2023-12-16T21:14:49.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.622+0000 7f896e59c640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f895006ae40 0x7f895006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:49.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.623+0000 7f896e59c640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f895006ae40 0x7f895006d300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f897015b610 tx=0x7f8958009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:49.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.626+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8960054670 con 0x7f8970072390 2023-12-16T21:14:49.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.816+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f89700ad640 con 0x7f8970072390 2023-12-16T21:14:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.816+0000 7f89677fe640 1 -- 172.21.15.46:0/272308088 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8960001b60 con 0x7f8970072390 2023-12-16T21:14:49.817 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:49.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f895006ae40 msgr2=0x7f895006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f895006ae40 0x7f895006d300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f897015b610 tx=0x7f8958009190 comp rx=0 tx=0).stop 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 msgr2=0x7f89700ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 0x7f89700ac1e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f896000caa0 tx=0x7f896000cf60 comp rx=0 tx=0).stop 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 shutdown_connections 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f895006ae40 0x7f895006d300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8970158cd0 0x7f8970145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f897015d2b0 0x7f89700ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 --2- 172.21.15.46:0/272308088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8970072390 0x7f89700ac1e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 >> 172.21.15.46:0/272308088 conn(0x7f89700b4720 msgr2=0x7f89700ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 shutdown_connections 2023-12-16T21:14:49.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:49.818+0000 7f8975161640 1 -- 172.21.15.46:0/272308088 wait complete. 2023-12-16T21:14:49.829 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:14:49.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:49.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:49.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[104135]: fuse finished with error 0 and tester_r 0 2023-12-16T21:14:49.860 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:49.943 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:14:49.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:49.944 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:14:49.959 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:14:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.105+0000 7f4d3d797640 1 Processor -- start 2023-12-16T21:14:50.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.105+0000 7f4d3d797640 1 -- start start 2023-12-16T21:14:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d3d797640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d38143490 0x7f4d380b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d3d797640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4d3815e730 0x7f4d380b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d3d797640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3815a4e0 0x7f4d3815a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d3d797640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3815b520 con 0x7f4d38143490 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d3d797640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d38143870 con 0x7f4d3815e730 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d3d797640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3815e2a0 con 0x7f4d3815a4e0 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d36ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d38143490 0x7f4d380b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d36ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4d38143490 0x7f4d380b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47042/0 (socket says 172.21.15.46:47042) 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.106+0000 7f4d36ffd640 1 -- 172.21.15.46:0/4072034324 learned_addr learned my addr 172.21.15.46:0/4072034324 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d377fe640 1 --2- 172.21.15.46:0/4072034324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3815a4e0 0x7f4d3815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d367fc640 1 --2- 172.21.15.46:0/4072034324 >> v2:172.21.15.121:3300/0 conn(0x7f4d3815e730 0x7f4d380b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d36ffd640 1 -- 172.21.15.46:0/4072034324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3815a4e0 msgr2=0x7f4d3815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/4072034324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3815a4e0 0x7f4d3815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d36ffd640 1 -- 172.21.15.46:0/4072034324 >> v2:172.21.15.121:3300/0 conn(0x7f4d3815e730 msgr2=0x7f4d380b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/4072034324 >> v2:172.21.15.121:3300/0 conn(0x7f4d3815e730 0x7f4d380b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d36ffd640 1 -- 172.21.15.46:0/4072034324 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3815b240 con 0x7f4d38143490 2023-12-16T21:14:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d367fc640 1 --2- 172.21.15.46:0/4072034324 >> v2:172.21.15.121:3300/0 conn(0x7f4d3815e730 0x7f4d380b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.107+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/4072034324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d38143490 0x7f4d380b0cf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4d380aeca0 tx=0x7f4d3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2954de6660576ad3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d35ffb640 1 -- 172.21.15.46:0/4072034324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30031870 con 0x7f4d38143490 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d35ffb640 1 -- 172.21.15.46:0/4072034324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d300319d0 con 0x7f4d38143490 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d35ffb640 1 -- 172.21.15.46:0/4072034324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30031d60 con 0x7f4d38143490 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 -- 172.21.15.46:0/4072034324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d38143490 msgr2=0x7f4d380b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 --2- 172.21.15.46:0/4072034324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d38143490 0x7f4d380b0cf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f4d380aeca0 tx=0x7f4d3002b910 comp rx=0 tx=0).stop 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 -- 172.21.15.46:0/4072034324 shutdown_connections 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 --2- 172.21.15.46:0/4072034324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3815a4e0 0x7f4d3815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 --2- 172.21.15.46:0/4072034324 >> v2:172.21.15.121:3300/0 conn(0x7f4d3815e730 0x7f4d380b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 --2- 172.21.15.46:0/4072034324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d38143490 0x7f4d380b0cf0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 -- 172.21.15.46:0/4072034324 >> 172.21.15.46:0/4072034324 conn(0x7f4d3806a4d0 msgr2=0x7f4d3806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 -- 172.21.15.46:0/4072034324 shutdown_connections 2023-12-16T21:14:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.108+0000 7f4d3d797640 1 -- 172.21.15.46:0/4072034324 wait complete. 2023-12-16T21:14:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 Processor -- start 2023-12-16T21:14:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 -- start start 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d38143490 0x7f4d380cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3815a4e0 0x7f4d380ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 0x7f4d380ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3813fcf0 con 0x7f4d3815e730 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380ae220 con 0x7f4d3815a4e0 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d3d797640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3812e840 con 0x7f4d38143490 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 0x7f4d380ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d377fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 0x7f4d380ad830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:47048/0 (socket says 172.21.15.46:47048) 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.110+0000 7f4d377fe640 1 -- 172.21.15.46:0/3049983114 learned_addr learned my addr 172.21.15.46:0/3049983114 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d367fc640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3815a4e0 0x7f4d380ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d38143490 0x7f4d380cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d377fe640 1 -- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d38143490 msgr2=0x7f4d380cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d377fe640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d38143490 0x7f4d380cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d377fe640 1 -- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3815a4e0 msgr2=0x7f4d380ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d377fe640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3815a4e0 0x7f4d380ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d377fe640 1 -- 172.21.15.46:0/3049983114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d300094d0 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d38143490 0x7f4d380cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d377fe640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 0x7f4d380ad830 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4d28007e40 tx=0x7f4d2800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28014620 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2800ee90 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d380adf40 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d280149e0 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.111+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d381e4d50 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.112+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3815a970 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.113+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4d28014b40 con 0x7f4d3815e730 2023-12-16T21:14:50.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.113+0000 7f4d1f7fe640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4d2406ae40 0x7f4d2406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.113+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f4d2808a120 con 0x7f4d3815e730 2023-12-16T21:14:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.113+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4d2406ae40 0x7f4d2406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:50.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.114+0000 7f4d36ffd640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4d2406ae40 0x7f4d2406d300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4d30036000 tx=0x7f4d3003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:50.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.117+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4d280550d0 con 0x7f4d3815e730 2023-12-16T21:14:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.329+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4d38069400 con 0x7f4d3815e730 2023-12-16T21:14:50.330 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.329+0000 7f4d1f7fe640 1 -- 172.21.15.46:0/3049983114 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+82 (secure 0 0 0) 0x7f4d280586f0 con 0x7f4d3815e730 2023-12-16T21:14:50.330 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4d2406ae40 msgr2=0x7f4d2406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4d2406ae40 0x7f4d2406d300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4d30036000 tx=0x7f4d3003c040 comp rx=0 tx=0).stop 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 msgr2=0x7f4d380ad830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 0x7f4d380ad830 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4d28007e40 tx=0x7f4d2800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 shutdown_connections 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4d2406ae40 0x7f4d2406d300 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d3815e730 0x7f4d380ad830 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3815a4e0 0x7f4d380ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 --2- 172.21.15.46:0/3049983114 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d38143490 0x7f4d380cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:50.332 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 >> 172.21.15.46:0/3049983114 conn(0x7f4d3806a4d0 msgr2=0x7f4d380aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 shutdown_connections 2023-12-16T21:14:50.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:50.331+0000 7f4d3d797640 1 -- 172.21.15.46:0/3049983114 wait complete. 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:14:50.343 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:14:50.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.343 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:14:50.359 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:14:50.359 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:14:50.359 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:14:50.359 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.359 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42594sec preferred_lft 42594sec 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.360 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:50.361 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::20ee:6cff:fee1:5615/64 scope link 2023-12-16T21:14:50.362 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:14:50.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.362 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:14:50.428 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:14:50.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.430 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:14:50.486 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:14:50.486 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:14:50.486 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:14:50.486 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:14:50.487 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:14:50.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.487 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:14:50.487 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:14:50.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:14:50.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:14:50.487 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:14:50.487 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:14:50.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.594 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:14:50.609 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:14:50.610 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:14:50.610 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:14:50.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:14:50.681 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:14:50.681 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:14:50.681 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:50.737 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:14:50.737 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:14:50.737 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:50.792 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:14:50.863 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:14:50.919 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:14:50.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:14:50.919 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:14:50.919 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:14:50.920 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:14:50.921 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:50.922 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:50.922 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:50.922 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:50.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:50.922 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:14:50.974 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:14:50.974 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:14:51.017 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:14:51.047 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:14:51.103 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:14:51.104 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:14:51.105 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:14:51.106 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:14:51.106 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:51.106 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:14:51.106 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:51.106 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:14:51.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:51.106 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:14:51.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:14:51.109+0000 7fb99ee3e480 -1 init, newargv = 0x55674814d670 newargc=15 2023-12-16T21:14:51.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[105369]: starting ceph client 2023-12-16T21:14:51.114 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[105369]: starting fuse 2023-12-16T21:14:51.124 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:14:51.125 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:14:51.125 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:14:51.126 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:14:51.249 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.105369.asok 2023-12-16T21:14:51.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:14:51.260 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.105369.asok 2023-12-16T21:14:51.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:51.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.105369.asok status 2023-12-16T21:14:51.402 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:14:51.402 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:14:51.402 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:14:51.402 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:14:51.402 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:14:51.402 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "105369", 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5568, 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5568 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:14:51.403 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2657832026 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2657832026 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5568 192.168.0.1:0/2657832026", 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2657832026", 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 14, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 37, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:14:51.404 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:14:51.405 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:14:51.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:51.410 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:14:51.411 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:51.427 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:14:51.427 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:51.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:51.427 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:51.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:51.499 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2023-12-16T21:14:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:14:51.562 INFO:teuthology.orchestra.run.smithi046.stderr:setfattr: layoutfile: Permission denied 2023-12-16T21:14:51.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:51.563 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2023-12-16T21:14:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:14:51.620 INFO:teuthology.orchestra.run.smithi046.stderr:setfattr: layoutdir: Permission denied 2023-12-16T21:14:51.620 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:51.620 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:51.676 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:51.822 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.822+0000 7f216b883640 1 Processor -- start 2023-12-16T21:14:51.823 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.822+0000 7f216b883640 1 -- start start 2023-12-16T21:14:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f216b883640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 0x7f21641426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f216b883640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2164161d20 con 0x7f2164160fb0 2023-12-16T21:14:51.825 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f216b883640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216415e760 con 0x7f216406ea50 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f216b883640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21640d1ee0 con 0x7f2164142270 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f2168e1e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2164160fb0 0x7f2164161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52614/0 (socket says 172.21.15.46:52614) 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f2168e1e640 1 -- 172.21.15.46:0/1728659592 learned_addr learned my addr 172.21.15.46:0/1728659592 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f2169e20640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 0x7f21641426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775028104 0 0) 0x7f2164161d20 con 0x7f2164160fb0 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.823+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2150003520 con 0x7f2164160fb0 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663146522 0 0) 0x7f216415e760 con 0x7f216406ea50 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2164161d20 con 0x7f216406ea50 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 213344095 0 0) 0x7f2150003520 con 0x7f2164160fb0 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 msgr2=0x7f21641426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 0x7f21641426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 >> v1:172.21.15.121:6789/0 conn(0x7f216406ea50 legacy=0x7f216406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216415d9a0 con 0x7f2164160fb0 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f2154001670 con 0x7f2164160fb0 2023-12-16T21:14:51.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 >> v1:172.21.15.46:6789/0 conn(0x7f2164160fb0 legacy=0x7f2164161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2150003b30 0x7f2150003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 0x7f21500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2150004d00 0x7f2150005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640d1ee0 con 0x7f2150004d00 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215000ba30 con 0x7f2150003b30 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f215000bba0 con 0x7f2164142270 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2169e20640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2150004d00 0x7f2150005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2168e1e640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 0x7f21500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f216961f640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2150003b30 0x7f2150003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2169e20640 1 -- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 msgr2=0x7f21500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2169e20640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2164142270 0x7f21500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2169e20640 1 -- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2150003b30 msgr2=0x7f2150003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2169e20640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2150003b30 0x7f2150003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.824+0000 7f2169e20640 1 -- 172.21.15.46:0/1728659592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2150003520 con 0x7f2150004d00 2023-12-16T21:14:51.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.825+0000 7f2169e20640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2150004d00 0x7f2150005180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f215c009fc0 tx=0x7f21500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f59cfbb4543c2fb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.825+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c02cc20 con 0x7f2150004d00 2023-12-16T21:14:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.825+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215c00f040 con 0x7f2150004d00 2023-12-16T21:14:51.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.825+0000 7f215bfff640 1 -- 172.21.15.46:0/1728659592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c034440 con 0x7f2150004d00 2023-12-16T21:14:51.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f216b883640 1 -- 172.21.15.46:0/1728659592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2150004d00 msgr2=0x7f2150005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:51.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f216b883640 1 --2- 172.21.15.46:0/1728659592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2150004d00 0x7f2150005180 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f215c009fc0 tx=0x7f21500057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:51.830 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f2169e20640 1 -- 172.21.15.46:0/1728659592 reap_dead start 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f216b883640 1 -- 172.21.15.46:0/1728659592 shutdown_connections 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f216b883640 1 -- 172.21.15.46:0/1728659592 >> 172.21.15.46:0/1728659592 conn(0x7f216406a6b0 msgr2=0x7f216406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f216b883640 1 -- 172.21.15.46:0/1728659592 shutdown_connections 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.826+0000 7f216b883640 1 -- 172.21.15.46:0/1728659592 wait complete. 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.827+0000 7f216b883640 1 Processor -- start 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.827+0000 7f216b883640 1 -- start start 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 0x7f2164157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f216413bf10 0x7f2164158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2164159160 0x7f21641f33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2164159610 con 0x7f216413bb30 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21640b1fd0 con 0x7f2164159160 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21641584a0 con 0x7f216413bf10 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 0x7f2164157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 0x7f2164157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39280/0 (socket says 172.21.15.46:39280) 2023-12-16T21:14:51.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 -- 172.21.15.46:0/1528747908 learned_addr learned my addr 172.21.15.46:0/1528747908 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f2169e20640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2164159160 0x7f21641f33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f2168e1e640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f216413bf10 0x7f2164158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 -- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f216413bf10 msgr2=0x7f2164158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f216413bf10 0x7f2164158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 -- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2164159160 msgr2=0x7f21641f33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2164159160 0x7f21641f33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 -- 172.21.15.46:0/1528747908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21640b16a0 con 0x7f216413bb30 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f2168e1e640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f216413bf10 0x7f2164158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216961f640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 0x7f2164157ec0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f216000f7a0 tx=0x7f216000fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216000fe50 con 0x7f216413bb30 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600052e0 con 0x7f216413bb30 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21640b1980 con 0x7f216413bb30 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.828+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f216000d970 con 0x7f216413bb30 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.829+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21640b2460 con 0x7f216413bb30 2023-12-16T21:14:51.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.830+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2164069400 con 0x7f216413bb30 2023-12-16T21:14:51.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.830+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f216000dad0 con 0x7f216413bb30 2023-12-16T21:14:51.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.831+0000 7f2159ffb640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f214806ae70 0x7f214806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:51.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.831+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2160089780 con 0x7f216413bb30 2023-12-16T21:14:51.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.831+0000 7f2168e1e640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f214806ae70 0x7f214806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:51.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.831+0000 7f2168e1e640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f214806ae70 0x7f214806d330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21640b4d00 tx=0x7f2154009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:51.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:51.835+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2160054730 con 0x7f216413bb30 2023-12-16T21:14:52.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.012+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2164141c20 con 0x7f216413bb30 2023-12-16T21:14:52.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.013+0000 7f2159ffb640 1 -- 172.21.15.46:0/1528747908 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7f2160057d50 con 0x7f216413bb30 2023-12-16T21:14:52.014 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:52.014 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f214806ae70 msgr2=0x7f214806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f214806ae70 0x7f214806d330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21640b4d00 tx=0x7f2154009040 comp rx=0 tx=0).stop 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 msgr2=0x7f2164157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 0x7f2164157ec0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f216000f7a0 tx=0x7f216000fc60 comp rx=0 tx=0).stop 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 shutdown_connections 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f214806ae70 0x7f214806d330 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2164159160 0x7f21641f33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f216413bf10 0x7f2164158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 --2- 172.21.15.46:0/1528747908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f216413bb30 0x7f2164157ec0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 >> 172.21.15.46:0/1528747908 conn(0x7f216406a6b0 msgr2=0x7f216415e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 shutdown_connections 2023-12-16T21:14:52.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.015+0000 7f216b883640 1 -- 172.21.15.46:0/1528747908 wait complete. 2023-12-16T21:14:52.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:52.027 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.136+0000 7f8dc20d3640 1 Processor -- start 2023-12-16T21:14:52.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.136+0000 7f8dc20d3640 1 -- start start 2023-12-16T21:14:52.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dc20d3640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8dbc143490 0x7f8dbc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dc20d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dbc15e2a0 con 0x7f8dbc15e730 2023-12-16T21:14:52.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dc20d3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dbc15b240 con 0x7f8dbc15abf0 2023-12-16T21:14:52.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dc20d3640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dbc15b520 con 0x7f8dbc143490 2023-12-16T21:14:52.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dbaffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8dbc15e730 0x7f8dbc15a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52630/0 (socket says 172.21.15.46:52630) 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dbaffd640 1 -- 172.21.15.46:0/2656161075 learned_addr learned my addr 172.21.15.46:0/2656161075 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dbb7fe640 1 --2- 172.21.15.46:0/2656161075 >> v2:172.21.15.121:3300/0 conn(0x7f8dbc143490 0x7f8dbc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316901522 0 0) 0x7f8dbc15e2a0 con 0x7f8dbc15e730 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da4003520 con 0x7f8dbc15e730 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4101348564 0 0) 0x7f8da4003520 con 0x7f8dbc15e730 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 >> v2:172.21.15.121:3300/0 conn(0x7f8dbc143490 msgr2=0x7f8dbc0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 --2- 172.21.15.46:0/2656161075 >> v2:172.21.15.121:3300/0 conn(0x7f8dbc143490 0x7f8dbc0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 >> v1:172.21.15.121:6789/0 conn(0x7f8dbc15abf0 legacy=0x7f8dbc0b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc15dc50 con 0x7f8dbc15e730 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8db0001670 con 0x7f8dbc15e730 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dbb7fe640 1 --2- 172.21.15.46:0/2656161075 >> v2:172.21.15.121:3300/0 conn(0x7f8dbc143490 0x7f8dbc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 >> v1:172.21.15.46:6789/0 conn(0x7f8dbc15e730 legacy=0x7f8dbc15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da4003970 0x7f8da4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da40047c0 0x7f8da4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da4005180 0x7f8da400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.137+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc15b520 con 0x7f8da40047c0 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da400c020 con 0x7f8da4005180 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da400c1e0 con 0x7f8da4003970 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da40047c0 0x7f8da4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbb7fe640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da4003970 0x7f8da4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbbfff640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da4005180 0x7f8da400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbaffd640 1 -- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da4003970 msgr2=0x7f8da4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da4003970 0x7f8da4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbaffd640 1 -- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da4005180 msgr2=0x7f8da400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da4005180 0x7f8da400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbaffd640 1 -- 172.21.15.46:0/2656161075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da4003520 con 0x7f8da40047c0 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.138+0000 7f8dbbfff640 1 -- 172.21.15.46:0/2656161075 reap_dead start 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da40047c0 0x7f8da4004c40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8db000c240 tx=0x7f8da40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd09e4e1f52be16 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0005ad0 con 0x7f8da40047c0 2023-12-16T21:14:52.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db0005c30 con 0x7f8da40047c0 2023-12-16T21:14:52.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dba7fc640 1 -- 172.21.15.46:0/2656161075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db002caf0 con 0x7f8da40047c0 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 -- 172.21.15.46:0/2656161075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da40047c0 msgr2=0x7f8da4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da40047c0 0x7f8da4004c40 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f8db000c240 tx=0x7f8da40057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 -- 172.21.15.46:0/2656161075 shutdown_connections 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/2656161075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8da40047c0 0x7f8da4004c40 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 -- 172.21.15.46:0/2656161075 >> 172.21.15.46:0/2656161075 conn(0x7f8dbc06a4d0 msgr2=0x7f8dbc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 -- 172.21.15.46:0/2656161075 shutdown_connections 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.139+0000 7f8dc20d3640 1 -- 172.21.15.46:0/2656161075 wait complete. 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 Processor -- start 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 -- start start 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da40047c0 0x7f8dbc13d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 0x7f8dbc13de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.145 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8dbc138700 0x7f8dbc138b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc139430 con 0x7f8dbc1381b0 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc139710 con 0x7f8da40047c0 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.141+0000 7f8dc20d3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc1399f0 con 0x7f8dbc138700 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 0x7f8dbc13de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 0x7f8dbc13de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39302/0 (socket says 172.21.15.46:39302) 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 -- 172.21.15.46:0/641681668 learned_addr learned my addr 172.21.15.46:0/641681668 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbb7fe640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da40047c0 0x7f8dbc13d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbbfff640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8dbc138700 0x7f8dbc138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 -- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8dbc138700 msgr2=0x7f8dbc138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8dbc138700 0x7f8dbc138b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 -- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da40047c0 msgr2=0x7f8dbc13d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da40047c0 0x7f8dbc13d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 -- 172.21.15.46:0/641681668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc139cd0 con 0x7f8dbc1381b0 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbbfff640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8dbc138700 0x7f8dbc138b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:52.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dbaffd640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 0x7f8dbc13de30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001930 tx=0x7f8db002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db003c040 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db002cd80 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dbc139fb0 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db0007290 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.142+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dbc13bc10 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.143+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dbc139b30 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.144+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8db0032070 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.144+0000 7f8da3fff640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8da806ae40 0x7f8da806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.144+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8db00aac40 con 0x7f8dbc1381b0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.144+0000 7f8dbb7fe640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8da806ae40 0x7f8da806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.145+0000 7f8dbb7fe640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8da806ae40 0x7f8da806d300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc050020 tx=0x7f8dac009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.148+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8dbc139b30 con 0x7f8dbc1381b0 2023-12-16T21:14:52.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.322+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8dbc139850 con 0x7f8dbc1381b0 2023-12-16T21:14:52.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.323+0000 7f8da3fff640 1 -- 172.21.15.46:0/641681668 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8dbc139850 con 0x7f8dbc1381b0 2023-12-16T21:14:52.324 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:52.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8da806ae40 msgr2=0x7f8da806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8da806ae40 0x7f8da806d300 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc050020 tx=0x7f8dac009190 comp rx=0 tx=0).stop 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 msgr2=0x7f8dbc13de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 0x7f8dbc13de30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8db0001930 tx=0x7f8db002e070 comp rx=0 tx=0).stop 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 shutdown_connections 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8da806ae40 0x7f8da806d300 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8dbc1381b0 0x7f8dbc13de30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8dbc138700 0x7f8dbc138b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 --2- 172.21.15.46:0/641681668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da40047c0 0x7f8dbc13d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 >> 172.21.15.46:0/641681668 conn(0x7f8dbc06a4d0 msgr2=0x7f8dbc0ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 shutdown_connections 2023-12-16T21:14:52.326 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.325+0000 7f8dc20d3640 1 -- 172.21.15.46:0/641681668 wait complete. 2023-12-16T21:14:52.336 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:52.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.445+0000 7fbe32a2d640 1 Processor -- start 2023-12-16T21:14:52.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.445+0000 7fbe32a2d640 1 -- start start 2023-12-16T21:14:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe32a2d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 0x7fbe2c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe32a2d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbe2c0b4370 0x7fbe2c0b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe32a2d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c071770 con 0x7fbe2c0ceb70 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe32a2d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2c0b4d20 con 0x7fbe2c0b4370 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe32a2d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c0ccb60 con 0x7fbe2c144980 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe2bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 0x7fbe2c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe30fca640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbe2c0b4370 0x7fbe2c0b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe2b7fe640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbe2c0ceb70 0x7fbe2c0cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:36206/0 (socket says 172.21.15.46:36206) 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe2bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 0x7fbe2c15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:44040/0 (socket says 172.21.15.46:44040) 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe30fca640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbe2c0b4370 0x7fbe2c0b47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:51174/0 (socket says 172.21.15.46:51174) 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.446+0000 7fbe2b7fe640 1 -- 172.21.15.46:0/1640735883 learned_addr learned my addr 172.21.15.46:0/1640735883 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe2affd640 1 -- 172.21.15.46:0/1640735883 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730799916 0 0) 0x7fbe2c071770 con 0x7fbe2c0ceb70 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe2affd640 1 -- 172.21.15.46:0/1640735883 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe18003520 con 0x7fbe2c0ceb70 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe30fca640 1 -- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 msgr2=0x7fbe2c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe30fca640 1 --2- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 0x7fbe2c15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe30fca640 1 -- 172.21.15.46:0/1640735883 >> v1:172.21.15.121:6789/0 conn(0x7fbe2c0ceb70 legacy=0x7fbe2c0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe30fca640 1 -- 172.21.15.46:0/1640735883 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c0ccdc0 con 0x7fbe2c0b4370 2023-12-16T21:14:52.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.447+0000 7fbe2bfff640 1 --2- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 0x7fbe2c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:14:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe30fca640 1 --2- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0b47e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009d90 tx=0x7fbe2402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bc05d6d156a78af server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe2affd640 1 -- 172.21.15.46:0/1640735883 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2402cc50 con 0x7fbe2c0b4370 2023-12-16T21:14:52.450 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe2affd640 1 -- 172.21.15.46:0/1640735883 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2402cdb0 con 0x7fbe2c0b4370 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe2affd640 1 -- 172.21.15.46:0/1640735883 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe24034640 con 0x7fbe2c0b4370 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 -- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 msgr2=0x7fbe2c0b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0b47e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009d90 tx=0x7fbe2402bea0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 -- 172.21.15.46:0/1640735883 shutdown_connections 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0b47e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/1640735883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c144980 0x7fbe2c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 -- 172.21.15.46:0/1640735883 >> 172.21.15.46:0/1640735883 conn(0x7fbe2c06a4d0 msgr2=0x7fbe2c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 -- 172.21.15.46:0/1640735883 shutdown_connections 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.448+0000 7fbe32a2d640 1 -- 172.21.15.46:0/1640735883 wait complete. 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.449+0000 7fbe32a2d640 1 Processor -- start 2023-12-16T21:14:52.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 -- start start 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c0ceb70 0x7fbe2c139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 0x7fbe2c13a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c0af2b0 con 0x7fbe2c144980 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c0af590 con 0x7fbe2c0b4370 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe32a2d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2c0af870 con 0x7fbe2c0ceb70 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe30fca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 0x7fbe2c13a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe30fca640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 0x7fbe2c13a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39304/0 (socket says 172.21.15.46:39304) 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.450+0000 7fbe30fca640 1 -- 172.21.15.46:0/4246724964 learned_addr learned my addr 172.21.15.46:0/4246724964 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe2bfff640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe2b7fe640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c0ceb70 0x7fbe2c139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe30fca640 1 -- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c0ceb70 msgr2=0x7fbe2c139a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe30fca640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c0ceb70 0x7fbe2c139a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe30fca640 1 -- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 msgr2=0x7fbe2c0aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe30fca640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe30fca640 1 -- 172.21.15.46:0/4246724964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14009400 con 0x7fbe2c144980 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe30fca640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 0x7fbe2c13a8a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009eb0 tx=0x7fbe24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2403e610 con 0x7fbe2c144980 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2403e770 con 0x7fbe2c144980 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe24009a70 con 0x7fbe2c144980 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2403eb60 con 0x7fbe2c144980 2023-12-16T21:14:52.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.451+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2c0afe30 con 0x7fbe2c144980 2023-12-16T21:14:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.452+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2c0b0020 con 0x7fbe2c144980 2023-12-16T21:14:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.453+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbe2403b050 con 0x7fbe2c144980 2023-12-16T21:14:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.453+0000 7fbe28ff9640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbe0c06ae40 0x7fbe0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.453+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fbe240ab8a0 con 0x7fbe2c144980 2023-12-16T21:14:52.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.453+0000 7fbe2bfff640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbe0c06ae40 0x7fbe0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.457 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.454+0000 7fbe2bfff640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbe0c06ae40 0x7fbe0c06d300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009cd0 tx=0x7fbe14009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.458 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.457+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fbe24076870 con 0x7fbe2c144980 2023-12-16T21:14:52.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.636+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe2c069400 con 0x7fbe2c144980 2023-12-16T21:14:52.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.636+0000 7fbe28ff9640 1 -- 172.21.15.46:0/4246724964 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe24039020 con 0x7fbe2c144980 2023-12-16T21:14:52.637 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbe0c06ae40 msgr2=0x7fbe0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbe0c06ae40 0x7fbe0c06d300 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe14009cd0 tx=0x7fbe14009190 comp rx=0 tx=0).stop 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 msgr2=0x7fbe2c13a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 0x7fbe2c13a8a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009eb0 tx=0x7fbe24004a40 comp rx=0 tx=0).stop 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 shutdown_connections 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbe0c06ae40 0x7fbe0c06d300 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbe2c144980 0x7fbe2c13a8a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe2c0ceb70 0x7fbe2c139a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 --2- 172.21.15.46:0/4246724964 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe2c0b4370 0x7fbe2c0aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 >> 172.21.15.46:0/4246724964 conn(0x7fbe2c06a4d0 msgr2=0x7fbe2c15e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 shutdown_connections 2023-12-16T21:14:52.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.638+0000 7fbe32a2d640 1 -- 172.21.15.46:0/4246724964 wait complete. 2023-12-16T21:14:52.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:52.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.758+0000 7f8fd916f640 1 Processor -- start 2023-12-16T21:14:52.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.758+0000 7f8fd916f640 1 -- start start 2023-12-16T21:14:52.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd916f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8fd4160fb0 0x7f8fd4161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd916f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd4161d20 con 0x7f8fd40d0ed0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd916f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd415e760 con 0x7f8fd406ea50 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd916f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd415d9a0 con 0x7f8fd4160fb0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd359e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8fd40d0ed0 0x7f8fd40d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52646/0 (socket says 172.21.15.46:52646) 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd359e640 1 -- 172.21.15.46:0/2248961756 learned_addr learned my addr 172.21.15.46:0/2248961756 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd259c640 1 --2- 172.21.15.46:0/2248961756 >> v2:172.21.15.121:3300/0 conn(0x7f8fd4160fb0 0x7f8fd4161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439034685 0 0) 0x7f8fd4161d20 con 0x7f8fd40d0ed0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.759+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fbc003520 con 0x7f8fd40d0ed0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896924558 0 0) 0x7f8fbc003520 con 0x7f8fd40d0ed0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 >> v2:172.21.15.121:3300/0 conn(0x7f8fd4160fb0 msgr2=0x7f8fd4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 --2- 172.21.15.46:0/2248961756 >> v2:172.21.15.121:3300/0 conn(0x7f8fd4160fb0 0x7f8fd4161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 >> v1:172.21.15.121:6789/0 conn(0x7f8fd406ea50 legacy=0x7f8fd406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd415dc80 con 0x7f8fd40d0ed0 2023-12-16T21:14:52.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8fc40019f0 con 0x7f8fd40d0ed0 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 >> v1:172.21.15.46:6789/0 conn(0x7f8fd40d0ed0 legacy=0x7f8fd40d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc003970 0x7f8fbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc0047c0 0x7f8fbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd415d9a0 con 0x7f8fbc005180 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc00c120 con 0x7f8fbc0047c0 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc00c2e0 con 0x7f8fbc003970 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd359e640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd2d9d640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc003970 0x7f8fbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd259c640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc0047c0 0x7f8fbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd359e640 1 -- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc003970 msgr2=0x7f8fbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd359e640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc003970 0x7f8fbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd359e640 1 -- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc0047c0 msgr2=0x7f8fbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:52.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd359e640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc0047c0 0x7f8fbc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.760+0000 7f8fd359e640 1 -- 172.21.15.46:0/2248961756 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc003520 con 0x7f8fbc005180 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.761+0000 7f8fd359e640 1 -- 172.21.15.46:0/2248961756 reap_dead start 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.761+0000 7f8fd359e640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fbc00b980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400a830 tx=0x7f8fbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e7cd849b2ee0dc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.761+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40072d0 con 0x7f8fbc005180 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.761+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc402f070 con 0x7f8fbc005180 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.761+0000 7f8fd1d9b640 1 -- 172.21.15.46:0/2248961756 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40076b0 con 0x7f8fbc005180 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 -- 172.21.15.46:0/2248961756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 msgr2=0x7f8fbc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fbc00b980 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400a830 tx=0x7f8fbc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 -- 172.21.15.46:0/2248961756 shutdown_connections 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 --2- 172.21.15.46:0/2248961756 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fbc00b980 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 -- 172.21.15.46:0/2248961756 >> 172.21.15.46:0/2248961756 conn(0x7f8fd406a6b0 msgr2=0x7f8fd406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 -- 172.21.15.46:0/2248961756 shutdown_connections 2023-12-16T21:14:52.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.762+0000 7f8fd916f640 1 -- 172.21.15.46:0/2248961756 wait complete. 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.763+0000 7f8fd916f640 1 Processor -- start 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.763+0000 7f8fd916f640 1 -- start start 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd916f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fd4156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd916f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd40cae10 0x7f8fd40cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd916f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd41588f0 0x7f8fd40b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd916f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41584d0 con 0x7f8fbc005180 2023-12-16T21:14:52.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd916f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd41578e0 con 0x7f8fd41588f0 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd916f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4157b30 con 0x7f8fd40cae10 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fd4156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fd4156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39324/0 (socket says 172.21.15.46:39324) 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 -- 172.21.15.46:0/1531275603 learned_addr learned my addr 172.21.15.46:0/1531275603 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd359e640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd41588f0 0x7f8fd40b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd259c640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd40cae10 0x7f8fd40cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 -- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd40cae10 msgr2=0x7f8fd40cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd40cae10 0x7f8fd40cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 -- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd41588f0 msgr2=0x7f8fd40b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd41588f0 0x7f8fd40b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 -- 172.21.15.46:0/1531275603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd4157e10 con 0x7f8fbc005180 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd359e640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd41588f0 0x7f8fd40b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.764+0000 7f8fd2d9d640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fd4156590 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00f5f0 tx=0x7f8fcc00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.765+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0052e0 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.765+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc005440 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.765+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc003810 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.765+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd41580f0 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.765+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd40cb910 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.766+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd415c410 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.766+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8fcc003970 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.767+0000 7f8fbb7fe640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fc006ae40 0x7f8fc006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.767+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8fcc0895a0 con 0x7f8fbc005180 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.767+0000 7f8fd259c640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fc006ae40 0x7f8fc006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:52.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.767+0000 7f8fd259c640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fc006ae40 0x7f8fc006d300 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4159140 tx=0x7f8fc8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:52.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.771+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8fcc054550 con 0x7f8fbc005180 2023-12-16T21:14:52.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.947+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8fd4158740 con 0x7f8fbc005180 2023-12-16T21:14:52.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.947+0000 7f8fbb7fe640 1 -- 172.21.15.46:0/1531275603 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8fcc057b70 con 0x7f8fbc005180 2023-12-16T21:14:52.948 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:52.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fc006ae40 msgr2=0x7f8fc006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fc006ae40 0x7f8fc006d300 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4159140 tx=0x7f8fc8008040 comp rx=0 tx=0).stop 2023-12-16T21:14:52.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 msgr2=0x7f8fd4156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:52.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fd4156590 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00f5f0 tx=0x7f8fcc00fab0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 shutdown_connections 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fc006ae40 0x7f8fc006d300 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd41588f0 0x7f8fd40b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd40cae10 0x7f8fd40cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 --2- 172.21.15.46:0/1531275603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fbc005180 0x7f8fd4156590 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 >> 172.21.15.46:0/1531275603 conn(0x7f8fd406a6b0 msgr2=0x7f8fd40b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 shutdown_connections 2023-12-16T21:14:52.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:52.949+0000 7f8fd916f640 1 -- 172.21.15.46:0/1531275603 wait complete. 2023-12-16T21:14:52.961 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:14:53.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.070+0000 7f2a67761640 1 Processor -- start 2023-12-16T21:14:53.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.070+0000 7f2a67761640 1 -- start start 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a67761640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a601618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a67761640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a601406f0 con 0x7f2a60140310 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a67761640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a60143960 con 0x7f2a6006abe0 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a67761640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60161e20 con 0x7f2a601409d0 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a654fd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a60140310 0x7f2a6006a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52656/0 (socket says 172.21.15.46:52656) 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a654fd640 1 -- 172.21.15.46:0/2011838254 learned_addr learned my addr 172.21.15.46:0/2011838254 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.071+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222049307 0 0) 0x7f2a601406f0 con 0x7f2a60140310 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a48003520 con 0x7f2a60140310 2023-12-16T21:14:53.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a65cfe640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a601618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935476519 0 0) 0x7f2a60143960 con 0x7f2a6006abe0 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a601406f0 con 0x7f2a6006abe0 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15415039 0 0) 0x7f2a48003520 con 0x7f2a60140310 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 msgr2=0x7f2a601618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a601618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 >> v1:172.21.15.121:6789/0 conn(0x7f2a6006abe0 legacy=0x7f2a6006efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60142de0 con 0x7f2a60140310 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f2a580015f0 con 0x7f2a60140310 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 >> v1:172.21.15.46:6789/0 conn(0x7f2a60140310 legacy=0x7f2a6006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a48003b30 0x7f2a48003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a480047c0 0x7f2a48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a480051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60161e20 con 0x7f2a480047c0 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4800bb10 con 0x7f2a48003b30 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4800bcd0 con 0x7f2a601409d0 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a480047c0 0x7f2a48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a654fd640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a48003b30 0x7f2a48003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a65cfe640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a480051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a64cfc640 1 -- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 msgr2=0x7f2a480051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a480051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a64cfc640 1 -- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a48003b30 msgr2=0x7f2a48003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a48003b30 0x7f2a48003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a64cfc640 1 -- 172.21.15.46:0/2011838254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a48003520 con 0x7f2a480047c0 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.072+0000 7f2a65cfe640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a601409d0 0x7f2a480051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.073+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a480047c0 0x7f2a48004c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2a540018d0 tx=0x7f2a480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa128fbaa903068 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.073+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a540052d0 con 0x7f2a480047c0 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.073+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a54005430 con 0x7f2a480047c0 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.073+0000 7f2a5ffff640 1 -- 172.21.15.46:0/2011838254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54003810 con 0x7f2a480047c0 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a67761640 1 -- 172.21.15.46:0/2011838254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a480047c0 msgr2=0x7f2a48004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a67761640 1 --2- 172.21.15.46:0/2011838254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a480047c0 0x7f2a48004c40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f2a540018d0 tx=0x7f2a480057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a65cfe640 1 -- 172.21.15.46:0/2011838254 reap_dead start 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a67761640 1 -- 172.21.15.46:0/2011838254 shutdown_connections 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a67761640 1 -- 172.21.15.46:0/2011838254 >> 172.21.15.46:0/2011838254 conn(0x7f2a600b1e50 msgr2=0x7f2a600b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a67761640 1 -- 172.21.15.46:0/2011838254 shutdown_connections 2023-12-16T21:14:53.077 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.074+0000 7f2a67761640 1 -- 172.21.15.46:0/2011838254 wait complete. 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.075+0000 7f2a67761640 1 Processor -- start 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.075+0000 7f2a67761640 1 -- start start 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a67761640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480047c0 0x7f2a6013d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a67761640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 0x7f2a60145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a67761640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a60146050 0x7f2a600aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a67761640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6013d820 con 0x7f2a601f3d80 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a67761640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60160180 con 0x7f2a60146050 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a67761640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60160460 con 0x7f2a480047c0 2023-12-16T21:14:53.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 0x7f2a60145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 0x7f2a60145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39346/0 (socket says 172.21.15.46:39346) 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 -- 172.21.15.46:0/1731525844 learned_addr learned my addr 172.21.15.46:0/1731525844 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a65cfe640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a60146050 0x7f2a600aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 -- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480047c0 msgr2=0x7f2a6013d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480047c0 0x7f2a6013d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 -- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a60146050 msgr2=0x7f2a600aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a60146050 0x7f2a600aebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.076+0000 7f2a64cfc640 1 -- 172.21.15.46:0/1731525844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a60160740 con 0x7f2a601f3d80 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.077+0000 7f2a64cfc640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 0x7f2a60145b10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2a54005040 tx=0x7f2a540012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.077+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54007450 con 0x7f2a601f3d80 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.077+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a540075b0 con 0x7f2a601f3d80 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.077+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a601f3860 con 0x7f2a601f3d80 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.077+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54007720 con 0x7f2a601f3d80 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.077+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6015fd30 con 0x7f2a601f3d80 2023-12-16T21:14:53.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.078+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a60161030 con 0x7f2a601f3d80 2023-12-16T21:14:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.078+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2a540037f0 con 0x7f2a601f3d80 2023-12-16T21:14:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.079+0000 7f2a5dffb640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a4c06ae40 0x7f2a4c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.079+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2a540ab6e0 con 0x7f2a601f3d80 2023-12-16T21:14:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.079+0000 7f2a654fd640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a4c06ae40 0x7f2a4c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.080+0000 7f2a654fd640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a4c06ae40 0x7f2a4c06d300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2a58001ae0 tx=0x7f2a580019f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.083+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2a540766b0 con 0x7f2a601f3d80 2023-12-16T21:14:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.257+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f2a60069400 con 0x7f2a601f3d80 2023-12-16T21:14:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.258+0000 7f2a5dffb640 1 -- 172.21.15.46:0/1731525844 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7f2a5404b020 con 0x7f2a601f3d80 2023-12-16T21:14:53.258 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a4c06ae40 msgr2=0x7f2a4c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a4c06ae40 0x7f2a4c06d300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2a58001ae0 tx=0x7f2a580019f0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 msgr2=0x7f2a60145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 0x7f2a60145b10 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f2a54005040 tx=0x7f2a540012d0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 shutdown_connections 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a4c06ae40 0x7f2a4c06d300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a601f3d80 0x7f2a60145b10 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a60146050 0x7f2a600aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.259+0000 7f2a67761640 1 --2- 172.21.15.46:0/1731525844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480047c0 0x7f2a6013d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.260+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 >> 172.21.15.46:0/1731525844 conn(0x7f2a600b1e50 msgr2=0x7f2a600b4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.260+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 shutdown_connections 2023-12-16T21:14:53.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.260+0000 7f2a67761640 1 -- 172.21.15.46:0/1731525844 wait complete. 2023-12-16T21:14:53.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.378+0000 7f3fda343640 1 Processor -- start 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.378+0000 7f3fda343640 1 -- start start 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fda343640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 0x7f3fd41426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fda343640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd4161d20 con 0x7f3fd4160fb0 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fda343640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd415e760 con 0x7f3fd406ea50 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fda343640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd40d1ee0 con 0x7f3fd4142270 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd37fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3fd4160fb0 0x7f3fd4161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52660/0 (socket says 172.21.15.46:52660) 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd37fe640 1 -- 172.21.15.46:0/1702978194 learned_addr learned my addr 172.21.15.46:0/1702978194 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd88e0640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 0x7f3fd41426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607516917 0 0) 0x7f3fd4161d20 con 0x7f3fd4160fb0 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fbc003520 con 0x7f3fd4160fb0 2023-12-16T21:14:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1255163607 0 0) 0x7f3fbc003520 con 0x7f3fd4160fb0 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 msgr2=0x7f3fd41426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 0x7f3fd41426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 >> v1:172.21.15.121:6789/0 conn(0x7f3fd406ea50 legacy=0x7f3fd406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd415d9a0 con 0x7f3fd4160fb0 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f3fc8001670 con 0x7f3fd4160fb0 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 >> v1:172.21.15.46:6789/0 conn(0x7f3fd4160fb0 legacy=0x7f3fd4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0039a0 0x7f3fbc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 0x7f3fbc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd40d1ee0 con 0x7f3fbc0047c0 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc00ba30 con 0x7f3fbc0039a0 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.379+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc00bbf0 con 0x7f3fd4142270 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd3fff640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0039a0 0x7f3fbc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd88e0640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 0x7f3fbc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd37fe640 1 -- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 msgr2=0x7f3fbc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd4142270 0x7f3fbc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd37fe640 1 -- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0039a0 msgr2=0x7f3fbc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0039a0 0x7f3fbc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.380+0000 7f3fd37fe640 1 -- 172.21.15.46:0/1702978194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc003520 con 0x7f3fbc0047c0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fbc004c40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8001eb0 tx=0x7f3fbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee57277e7e6aecae server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8005ad0 con 0x7f3fbc0047c0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc8005c30 con 0x7f3fbc0047c0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fd2ffd640 1 -- 172.21.15.46:0/1702978194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc802c850 con 0x7f3fbc0047c0 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fda343640 1 -- 172.21.15.46:0/1702978194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 msgr2=0x7f3fbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fda343640 1 --2- 172.21.15.46:0/1702978194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fbc004c40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8001eb0 tx=0x7f3fbc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fd88e0640 1 -- 172.21.15.46:0/1702978194 reap_dead start 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fda343640 1 -- 172.21.15.46:0/1702978194 shutdown_connections 2023-12-16T21:14:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fda343640 1 -- 172.21.15.46:0/1702978194 >> 172.21.15.46:0/1702978194 conn(0x7f3fd406a6b0 msgr2=0x7f3fd406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fda343640 1 -- 172.21.15.46:0/1702978194 shutdown_connections 2023-12-16T21:14:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.381+0000 7f3fda343640 1 -- 172.21.15.46:0/1702978194 wait complete. 2023-12-16T21:14:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.382+0000 7f3fda343640 1 Processor -- start 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 -- start start 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fd40b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd41e7ee0 0x7f3fd41e8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fd415a5b0 0x7f3fd415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd41e89a0 con 0x7f3fbc0047c0 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd41e8c80 con 0x7f3fd415a5b0 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fda343640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd41e8f60 con 0x7f3fd41e7ee0 2023-12-16T21:14:53.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fd40b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fd3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fd40b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39358/0 (socket says 172.21.15.46:39358) 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.383+0000 7f3fd3fff640 1 -- 172.21.15.46:0/553573902 learned_addr learned my addr 172.21.15.46:0/553573902 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd88e0640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fd415a5b0 0x7f3fd415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd41e7ee0 0x7f3fd41e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd3fff640 1 -- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd41e7ee0 msgr2=0x7f3fd41e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd3fff640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd41e7ee0 0x7f3fd41e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd3fff640 1 -- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fd415a5b0 msgr2=0x7f3fd415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd3fff640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fd415a5b0 0x7f3fd415aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd3fff640 1 -- 172.21.15.46:0/553573902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd41e9240 con 0x7f3fbc0047c0 2023-12-16T21:14:53.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd88e0640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fd415a5b0 0x7f3fd415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd41e7ee0 0x7f3fd41e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd3fff640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fd40b34b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400f5f0 tx=0x7f3fc400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc40052e0 con 0x7f3fbc0047c0 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4005440 con 0x7f3fbc0047c0 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd41e9520 con 0x7f3fbc0047c0 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4003810 con 0x7f3fbc0047c0 2023-12-16T21:14:53.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.384+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd413f750 con 0x7f3fbc0047c0 2023-12-16T21:14:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.385+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd4069400 con 0x7f3fbc0047c0 2023-12-16T21:14:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.385+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3fc4003970 con 0x7f3fbc0047c0 2023-12-16T21:14:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.386+0000 7f3fd0ff9640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3fc006ae40 0x7f3fc006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.386+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f3fc4089700 con 0x7f3fbc0047c0 2023-12-16T21:14:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.386+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3fc006ae40 0x7f3fc006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.387+0000 7f3fd37fe640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3fc006ae40 0x7f3fc006d300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8034000 tx=0x7f3fc8037040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.390+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3fc40545a0 con 0x7f3fbc0047c0 2023-12-16T21:14:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.579+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3fd4070d30 con 0x7f3fbc0047c0 2023-12-16T21:14:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.579+0000 7f3fd0ff9640 1 -- 172.21.15.46:0/553573902 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3fc4057bc0 con 0x7f3fbc0047c0 2023-12-16T21:14:53.580 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3fc006ae40 msgr2=0x7f3fc006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3fc006ae40 0x7f3fc006d300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8034000 tx=0x7f3fc8037040 comp rx=0 tx=0).stop 2023-12-16T21:14:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 msgr2=0x7f3fd40b34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fd40b34b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400f5f0 tx=0x7f3fc400fab0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 shutdown_connections 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3fc006ae40 0x7f3fc006d300 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fd415a5b0 0x7f3fd415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3fd41e7ee0 0x7f3fd41e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 --2- 172.21.15.46:0/553573902 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3fbc0047c0 0x7f3fd40b34b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 >> 172.21.15.46:0/553573902 conn(0x7f3fd406a6b0 msgr2=0x7f3fd406ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.581+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 shutdown_connections 2023-12-16T21:14:53.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.582+0000 7f3fda343640 1 -- 172.21.15.46:0/553573902 wait complete. 2023-12-16T21:14:53.593 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:14:53.593 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:14:53.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:14:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.755+0000 7f346dc89640 1 Processor -- start 2023-12-16T21:14:53.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f346dc89640 1 -- start start 2023-12-16T21:14:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f346dc89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f346815d330 0x7f34680b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f346dc89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680b2b10 con 0x7f34680d3900 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f346dc89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680d1590 con 0x7f34680d2910 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f346dc89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680d1870 con 0x7f346815d330 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f34677fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f346815d330 0x7f34680b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f3467fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f34680d3900 0x7f34680b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52676/0 (socket says 172.21.15.46:52676) 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.756+0000 7f3467fff640 1 -- 172.21.15.46:0/1594232963 learned_addr learned my addr 172.21.15.46:0/1594232963 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.757+0000 7f34677fe640 1 -- 172.21.15.46:0/1594232963 >> v1:172.21.15.121:6789/0 conn(0x7f34680d2910 legacy=0x7f34680d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.757+0000 7f34677fe640 1 -- 172.21.15.46:0/1594232963 >> v1:172.21.15.46:6789/0 conn(0x7f34680d3900 legacy=0x7f34680b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.757+0000 7f34667fc640 1 -- 172.21.15.46:0/1594232963 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925158518 0 0) 0x7f34680b2b10 con 0x7f34680d3900 2023-12-16T21:14:53.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.757+0000 7f34677fe640 1 -- 172.21.15.46:0/1594232963 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680d1b50 con 0x7f346815d330 2023-12-16T21:14:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f34677fe640 1 --2- 172.21.15.46:0/1594232963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346815d330 0x7f34680b8b80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f34580097a0 tx=0x7f345802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164195d8fea3b435 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f34667fc640 1 -- 172.21.15.46:0/1594232963 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345802e070 con 0x7f346815d330 2023-12-16T21:14:53.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f34667fc640 1 -- 172.21.15.46:0/1594232963 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34580315a0 con 0x7f346815d330 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f34667fc640 1 -- 172.21.15.46:0/1594232963 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458031930 con 0x7f346815d330 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 -- 172.21.15.46:0/1594232963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346815d330 msgr2=0x7f34680b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 --2- 172.21.15.46:0/1594232963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346815d330 0x7f34680b8b80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f34580097a0 tx=0x7f345802b840 comp rx=0 tx=0).stop 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 -- 172.21.15.46:0/1594232963 shutdown_connections 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 --2- 172.21.15.46:0/1594232963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f346815d330 0x7f34680b8b80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 -- 172.21.15.46:0/1594232963 >> 172.21.15.46:0/1594232963 conn(0x7f3468072020 msgr2=0x7f3468072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 -- 172.21.15.46:0/1594232963 shutdown_connections 2023-12-16T21:14:53.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.758+0000 7f346dc89640 1 -- 172.21.15.46:0/1594232963 wait complete. 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.759+0000 7f346dc89640 1 Processor -- start 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 -- start start 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 0x7f346815f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34680d3900 0x7f346815ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f346815d330 0x7f3468160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681f6e00 con 0x7f34680d2910 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34681f61f0 con 0x7f34680d3900 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f346dc89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34681f6460 con 0x7f346815d330 2023-12-16T21:14:53.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.760+0000 7f34677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 0x7f346815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 0x7f346815f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39376/0 (socket says 172.21.15.46:39376) 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 -- 172.21.15.46:0/2775243736 learned_addr learned my addr 172.21.15.46:0/2775243736 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f3466ffd640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34680d3900 0x7f346815ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f3467fff640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f346815d330 0x7f3468160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 -- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f346815d330 msgr2=0x7f3468160490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f346815d330 0x7f3468160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 -- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34680d3900 msgr2=0x7f346815ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34680d3900 0x7f346815ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 -- 172.21.15.46:0/2775243736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009400 con 0x7f34680d2910 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f3466ffd640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34680d3900 0x7f346815ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f34677fe640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 0x7f346815f480 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f34580098c0 tx=0x7f34580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458004230 con 0x7f34680d2910 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3458004390 con 0x7f34680d2910 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34681f6740 con 0x7f34680d2910 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3458032880 con 0x7f34680d2910 2023-12-16T21:14:53.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.761+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346815f9c0 con 0x7f34680d2910 2023-12-16T21:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.762+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346815fb80 con 0x7f34680d2910 2023-12-16T21:14:53.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.763+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f34580329e0 con 0x7f34680d2910 2023-12-16T21:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.763+0000 7f344ffff640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f345406ae40 0x7f345406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.763+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f34580ab8e0 con 0x7f34680d2910 2023-12-16T21:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.763+0000 7f3466ffd640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f345406ae40 0x7f345406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:53.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.764+0000 7f3466ffd640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f345406ae40 0x7f345406d300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f345c003eb0 tx=0x7f345c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:53.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.767+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f346815fb80 con 0x7f34680d2910 2023-12-16T21:14:53.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.946+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f346815fb80 con 0x7f34680d2910 2023-12-16T21:14:53.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.947+0000 7f344ffff640 1 -- 172.21.15.46:0/2775243736 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7f346815fb80 con 0x7f34680d2910 2023-12-16T21:14:53.948 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:14:53.948 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f345406ae40 msgr2=0x7f345406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f345406ae40 0x7f345406d300 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f345c003eb0 tx=0x7f345c009040 comp rx=0 tx=0).stop 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 msgr2=0x7f346815f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 0x7f346815f480 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f34580098c0 tx=0x7f34580046c0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 shutdown_connections 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f345406ae40 0x7f345406d300 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f346815d330 0x7f3468160490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34680d3900 0x7f346815ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 --2- 172.21.15.46:0/2775243736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34680d2910 0x7f346815f480 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 >> 172.21.15.46:0/2775243736 conn(0x7f3468072020 msgr2=0x7f3468072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 shutdown_connections 2023-12-16T21:14:53.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:53.949+0000 7f346dc89640 1 -- 172.21.15.46:0/2775243736 wait complete. 2023-12-16T21:14:53.960 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:14:53.961 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:14:54.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.070+0000 7f15ef0d0640 1 Processor -- start 2023-12-16T21:14:54.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.070+0000 7f15ef0d0640 1 -- start start 2023-12-16T21:14:54.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15ef0d0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15e80d2910 0x7f15e80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15ef0d0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80b2b10 con 0x7f15e80d3730 2023-12-16T21:14:54.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15ef0d0640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e80d1590 con 0x7f15e815d330 2023-12-16T21:14:54.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15ef0d0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15e80d3bc0 con 0x7f15e80d2910 2023-12-16T21:14:54.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15e7fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f15e80d2910 0x7f15e80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15ed66d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f15e80d3730 0x7f15e80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52682/0 (socket says 172.21.15.46:52682) 2023-12-16T21:14:54.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.071+0000 7f15ed66d640 1 -- 172.21.15.46:0/4178619150 learned_addr learned my addr 172.21.15.46:0/4178619150 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:54.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.072+0000 7f15e7fff640 1 -- 172.21.15.46:0/4178619150 >> v1:172.21.15.121:6789/0 conn(0x7f15e815d330 legacy=0x7f15e80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.072+0000 7f15e7fff640 1 -- 172.21.15.46:0/4178619150 >> v1:172.21.15.46:6789/0 conn(0x7f15e80d3730 legacy=0x7f15e80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.072+0000 7f15e7fff640 1 -- 172.21.15.46:0/4178619150 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80d1870 con 0x7f15e80d2910 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.072+0000 7f15e7fff640 1 --2- 172.21.15.46:0/4178619150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e80d31f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f15d00097a0 tx=0x7f15d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5ed965dcabc0e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15e77fe640 1 -- 172.21.15.46:0/4178619150 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0031830 con 0x7f15e80d2910 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15e77fe640 1 -- 172.21.15.46:0/4178619150 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d0031990 con 0x7f15e80d2910 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15e77fe640 1 -- 172.21.15.46:0/4178619150 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d0031d20 con 0x7f15e80d2910 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 -- 172.21.15.46:0/4178619150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 msgr2=0x7f15e80d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/4178619150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e80d31f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f15d00097a0 tx=0x7f15d002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 -- 172.21.15.46:0/4178619150 shutdown_connections 2023-12-16T21:14:54.078 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/4178619150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e80d31f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 -- 172.21.15.46:0/4178619150 >> 172.21.15.46:0/4178619150 conn(0x7f15e8072020 msgr2=0x7f15e8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 -- 172.21.15.46:0/4178619150 shutdown_connections 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.073+0000 7f15ef0d0640 1 -- 172.21.15.46:0/4178619150 wait complete. 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.074+0000 7f15ef0d0640 1 Processor -- start 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 -- start start 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e81f3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15e80d3730 0x7f15e80cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15e815d330 0x7f15e80b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8159c00 con 0x7f15e80d2910 2023-12-16T21:14:54.079 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80cd6d0 con 0x7f15e815d330 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ef0d0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80c8920 con 0x7f15e80d3730 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.075+0000 7f15ece6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e81f3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e81f3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39400/0 (socket says 172.21.15.46:39400) 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 -- 172.21.15.46:0/469892950 learned_addr learned my addr 172.21.15.46:0/469892950 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ed66d640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15e815d330 0x7f15e80b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15e7fff640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15e80d3730 0x7f15e80cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 -- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15e80d3730 msgr2=0x7f15e80cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15e80d3730 0x7f15e80cce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 -- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15e815d330 msgr2=0x7f15e80b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15e815d330 0x7f15e80b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 -- 172.21.15.46:0/469892950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d0009400 con 0x7f15e80d2910 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ece6c640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e81f3010 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c730 tx=0x7f15e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e0003b70 con 0x7f15e80d2910 2023-12-16T21:14:54.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e0003cf0 con 0x7f15e80d2910 2023-12-16T21:14:54.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e80c7ff0 con 0x7f15e80d2910 2023-12-16T21:14:54.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e000aa90 con 0x7f15e80d2910 2023-12-16T21:14:54.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.076+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e80c85b0 con 0x7f15e80d2910 2023-12-16T21:14:54.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.077+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e8066da0 con 0x7f15e80d2910 2023-12-16T21:14:54.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.078+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f15e000abf0 con 0x7f15e80d2910 2023-12-16T21:14:54.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.078+0000 7f15e57fa640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f15c806ae70 0x7f15c806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.078+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f15e0089410 con 0x7f15e80d2910 2023-12-16T21:14:54.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.078+0000 7f15e7fff640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f15c806ae70 0x7f15c806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.079+0000 7f15e7fff640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f15c806ae70 0x7f15c806d330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f15d0036000 tx=0x7f15d003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.082+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f15e0054470 con 0x7f15e80d2910 2023-12-16T21:14:54.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.256+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15e8069400 con 0x7f15e80d2910 2023-12-16T21:14:54.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.256+0000 7f15e57fa640 1 -- 172.21.15.46:0/469892950 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f15e0003e60 con 0x7f15e80d2910 2023-12-16T21:14:54.257 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f15c806ae70 msgr2=0x7f15c806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f15c806ae70 0x7f15c806d330 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f15d0036000 tx=0x7f15d003c040 comp rx=0 tx=0).stop 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 msgr2=0x7f15e81f3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e81f3010 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f15e000c730 tx=0x7f15e000cbf0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 shutdown_connections 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f15c806ae70 0x7f15c806d330 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15e815d330 0x7f15e80b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f15e80d3730 0x7f15e80cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 --2- 172.21.15.46:0/469892950 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15e80d2910 0x7f15e81f3010 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 >> 172.21.15.46:0/469892950 conn(0x7f15e8072020 msgr2=0x7f15e8072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 shutdown_connections 2023-12-16T21:14:54.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.258+0000 7f15ef0d0640 1 -- 172.21.15.46:0/469892950 wait complete. 2023-12-16T21:14:54.270 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:14:54.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.379+0000 7f88de3b0640 1 Processor -- start 2023-12-16T21:14:54.380 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.379+0000 7f88de3b0640 1 -- start start 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88de3b0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f88d8150560 0x7f88d80b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88de3b0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88d80b3390 0x7f88d815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88de3b0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d80b49f0 con 0x7f88d8150940 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88de3b0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d80b3de0 con 0x7f88d80b3390 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88de3b0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d80b40c0 con 0x7f88d8150560 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f88d80b3390 0x7f88d815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88dc94d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88d8150940 0x7f88d80ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52692/0 (socket says 172.21.15.46:52692) 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88dc94d640 1 -- 172.21.15.46:0/485790355 learned_addr learned my addr 172.21.15.46:0/485790355 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:54.383 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d7fff640 1 --2- 172.21.15.46:0/485790355 >> v2:172.21.15.121:3300/0 conn(0x7f88d8150560 0x7f88d80b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d6ffd640 1 -- 172.21.15.46:0/485790355 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719703150 0 0) 0x7f88d80b49f0 con 0x7f88d8150940 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d77fe640 1 -- 172.21.15.46:0/485790355 >> v2:172.21.15.121:3300/0 conn(0x7f88d8150560 msgr2=0x7f88d80b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d77fe640 1 --2- 172.21.15.46:0/485790355 >> v2:172.21.15.121:3300/0 conn(0x7f88d8150560 0x7f88d80b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d77fe640 1 -- 172.21.15.46:0/485790355 >> v1:172.21.15.46:6789/0 conn(0x7f88d8150940 legacy=0x7f88d80ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.380+0000 7f88d77fe640 1 -- 172.21.15.46:0/485790355 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d80b43a0 con 0x7f88d80b3390 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.381+0000 7f88d77fe640 1 --2- 172.21.15.46:0/485790355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d815d200 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f88d8072840 tx=0x7f88c802cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd548f6b1fd89909 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.381+0000 7f88d6ffd640 1 -- 172.21.15.46:0/485790355 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c802db50 con 0x7f88d80b3390 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.381+0000 7f88d6ffd640 1 -- 172.21.15.46:0/485790355 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c802dcb0 con 0x7f88d80b3390 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.381+0000 7f88d6ffd640 1 -- 172.21.15.46:0/485790355 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c8035660 con 0x7f88d80b3390 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 -- 172.21.15.46:0/485790355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 msgr2=0x7f88d815d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 --2- 172.21.15.46:0/485790355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d815d200 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f88d8072840 tx=0x7f88c802cda0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 -- 172.21.15.46:0/485790355 shutdown_connections 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 --2- 172.21.15.46:0/485790355 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d815d200 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 --2- 172.21.15.46:0/485790355 >> v2:172.21.15.121:3300/0 conn(0x7f88d8150560 0x7f88d80b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 -- 172.21.15.46:0/485790355 >> 172.21.15.46:0/485790355 conn(0x7f88d806a4d0 msgr2=0x7f88d806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:54.384 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 -- 172.21.15.46:0/485790355 shutdown_connections 2023-12-16T21:14:54.385 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.382+0000 7f88de3b0640 1 -- 172.21.15.46:0/485790355 wait complete. 2023-12-16T21:14:54.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.383+0000 7f88de3b0640 1 Processor -- start 2023-12-16T21:14:54.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.383+0000 7f88de3b0640 1 -- start start 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88de3b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d8152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88de3b0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d8150560 0x7f88d81557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88de3b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d8155ce0 0x7f88d80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88de3b0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8156240 con 0x7f88d80b3390 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88de3b0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8156520 con 0x7f88d8150560 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88de3b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d8156800 con 0x7f88d8155ce0 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d8152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d8152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39416/0 (socket says 172.21.15.46:39416) 2023-12-16T21:14:54.388 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 -- 172.21.15.46:0/1999281434 learned_addr learned my addr 172.21.15.46:0/1999281434 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 -- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d8155ce0 msgr2=0x7f88d80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d8155ce0 0x7f88d80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d77fe640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d8150560 0x7f88d81557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 -- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d8150560 msgr2=0x7f88d81557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d8150560 0x7f88d81557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 -- 172.21.15.46:0/1999281434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c800a5a0 con 0x7f88d80b3390 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.384+0000 7f88d7fff640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d8152ac0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f88c000d800 tx=0x7f88c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.389 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.385+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000b6c0 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.385+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c000b820 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.385+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c000bbb0 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.385+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d8156ae0 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.385+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d8153000 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.386+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d8069400 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.386+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f88c000bd10 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.386+0000 7f88d4ff9640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88b806ae70 0x7f88b806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.387+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f88c0089e20 con 0x7f88d80b3390 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.387+0000 7f88d77fe640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88b806ae70 0x7f88b806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.387+0000 7f88d77fe640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88b806ae70 0x7f88b806d330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88c8037000 tx=0x7f88c8031000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.391+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f88c0054cb0 con 0x7f88d80b3390 2023-12-16T21:14:54.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.569+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88d80713f0 con 0x7f88d80b3390 2023-12-16T21:14:54.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.569+0000 7f88d4ff9640 1 -- 172.21.15.46:0/1999281434 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88c00582d0 con 0x7f88d80b3390 2023-12-16T21:14:54.570 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88b806ae70 msgr2=0x7f88b806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88b806ae70 0x7f88b806d330 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f88c8037000 tx=0x7f88c8031000 comp rx=0 tx=0).stop 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 msgr2=0x7f88d8152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d8152ac0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f88c000d800 tx=0x7f88c000dcc0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 shutdown_connections 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88b806ae70 0x7f88b806d330 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88d8155ce0 0x7f88d80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88d8150560 0x7f88d81557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 --2- 172.21.15.46:0/1999281434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88d80b3390 0x7f88d8152ac0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 >> 172.21.15.46:0/1999281434 conn(0x7f88d806a4d0 msgr2=0x7f88d8070600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 shutdown_connections 2023-12-16T21:14:54.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.571+0000 7f88de3b0640 1 -- 172.21.15.46:0/1999281434 wait complete. 2023-12-16T21:14:54.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:14:54.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.692+0000 7f8fc3fec640 1 Processor -- start 2023-12-16T21:14:54.692 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.692+0000 7f8fc3fec640 1 -- start start 2023-12-16T21:14:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc3fec640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8fbc06ea50 0x7f8fbc06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.695 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc3fec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 0x7f8fbc161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc3fec640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc161d20 con 0x7f8fbc0b3ff0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc3fec640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc0d1ee0 con 0x7f8fbc06ea50 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc3fec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc06f370 con 0x7f8fbc160fb0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc2589640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8fbc0b3ff0 0x7f8fbc0b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:52698/0 (socket says 172.21.15.46:52698) 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc2589640 1 -- 172.21.15.46:0/2203845110 learned_addr learned my addr 172.21.15.46:0/2203845110 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171456497 0 0) 0x7f8fbc161d20 con 0x7f8fbc0b3ff0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa8003520 con 0x7f8fbc0b3ff0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc1d88640 1 --2- 172.21.15.46:0/2203845110 >> v2:172.21.15.121:3300/0 conn(0x7f8fbc06ea50 0x7f8fbc06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc1587640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 0x7f8fbc161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2250177209 0 0) 0x7f8fa8003520 con 0x7f8fbc0b3ff0 2023-12-16T21:14:54.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 msgr2=0x7f8fbc161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 0x7f8fbc161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 >> v2:172.21.15.121:3300/0 conn(0x7f8fbc06ea50 msgr2=0x7f8fbc06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 --2- 172.21.15.46:0/2203845110 >> v2:172.21.15.121:3300/0 conn(0x7f8fbc06ea50 0x7f8fbc06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc15e760 con 0x7f8fbc0b3ff0 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8fb40019f0 con 0x7f8fbc0b3ff0 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 >> v1:172.21.15.46:6789/0 conn(0x7f8fbc0b3ff0 legacy=0x7f8fbc0b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 0x7f8fa8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.693+0000 7f8fc0d86640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fa8003ef0 0x7f8fa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc0d86640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0d1ee0 con 0x7f8fa8004d00 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc06f370 con 0x7f8fa8003ef0 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa800b980 con 0x7f8fbc160fb0 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc2589640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc1d88640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 0x7f8fa8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc1587640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fa8003ef0 0x7f8fa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc2589640 1 -- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 msgr2=0x7f8fa8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc2589640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc160fb0 0x7f8fa8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc2589640 1 -- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fa8003ef0 msgr2=0x7f8fa80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:54.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc2589640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fa8003ef0 0x7f8fa80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.694+0000 7f8fc2589640 1 -- 172.21.15.46:0/2203845110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa8003520 con 0x7f8fa8004d00 2023-12-16T21:14:54.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc2589640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fa8005180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400a860 tx=0x7f8fa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4841f29ea2388e08 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4034630 con 0x7f8fa8004d00 2023-12-16T21:14:54.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb402f070 con 0x7f8fa8004d00 2023-12-16T21:14:54.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc0d86640 1 -- 172.21.15.46:0/2203845110 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4034a30 con 0x7f8fa8004d00 2023-12-16T21:14:54.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2203845110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 msgr2=0x7f8fa8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2203845110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fa8005180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400a860 tx=0x7f8fa80057f0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc2589640 1 -- 172.21.15.46:0/2203845110 reap_dead start 2023-12-16T21:14:54.702 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2203845110 shutdown_connections 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.695+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2203845110 >> 172.21.15.46:0/2203845110 conn(0x7f8fbc06a6b0 msgr2=0x7f8fbc06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.696+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2203845110 shutdown_connections 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.696+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2203845110 wait complete. 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 Processor -- start 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 -- start start 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fbc150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc1583c0 0x7f8fbc06c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc135410 0x7f8fbc135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc152300 con 0x7f8fa8004d00 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc135dd0 con 0x7f8fbc135410 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.697+0000 7f8fc3fec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc13ffe0 con 0x7f8fbc1583c0 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fbc150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fbc150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39424/0 (socket says 172.21.15.46:39424) 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 -- 172.21.15.46:0/2405742586 learned_addr learned my addr 172.21.15.46:0/2405742586 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc2589640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc135410 0x7f8fbc135890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1587640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc1583c0 0x7f8fbc06c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 -- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc1583c0 msgr2=0x7f8fbc06c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc1583c0 0x7f8fbc06c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 -- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc135410 msgr2=0x7f8fbc135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc135410 0x7f8fbc135890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 -- 172.21.15.46:0/2405742586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc1516f0 con 0x7f8fa8004d00 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1587640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc1583c0 0x7f8fbc06c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc1d88640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fbc150ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8011aa0 tx=0x7f8fb8011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8009680 con 0x7f8fa8004d00 2023-12-16T21:14:54.704 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb800fb80 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc1519d0 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb8014440 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.698+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc151f90 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.699+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc069400 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.700+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8fb801d400 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.700+0000 7f8fb27fc640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fa006ae40 0x7f8fa006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.700+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8fb808a050 con 0x7f8fa8004d00 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.701+0000 7f8fc1587640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fa006ae40 0x7f8fa006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.701+0000 7f8fc1587640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fa006ae40 0x7f8fa006d300 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc1f65f0 tx=0x7f8fac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.705+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8fb8054f70 con 0x7f8fa8004d00 2023-12-16T21:14:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.879+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8fbc070a50 con 0x7f8fa8004d00 2023-12-16T21:14:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.880+0000 7f8fb27fc640 1 -- 172.21.15.46:0/2405742586 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8fb8058590 con 0x7f8fa8004d00 2023-12-16T21:14:54.881 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:14:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fa006ae40 msgr2=0x7f8fa006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fa006ae40 0x7f8fa006d300 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc1f65f0 tx=0x7f8fac008040 comp rx=0 tx=0).stop 2023-12-16T21:14:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 msgr2=0x7f8fbc150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fbc150ff0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8fb8011aa0 tx=0x7f8fb8011f60 comp rx=0 tx=0).stop 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 shutdown_connections 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8fa006ae40 0x7f8fa006d300 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fbc135410 0x7f8fbc135890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fbc1583c0 0x7f8fbc06c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 --2- 172.21.15.46:0/2405742586 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8fa8004d00 0x7f8fbc150ff0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 >> 172.21.15.46:0/2405742586 conn(0x7f8fbc06a6b0 msgr2=0x7f8fbc15bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 shutdown_connections 2023-12-16T21:14:54.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:54.882+0000 7f8fc3fec640 1 -- 172.21.15.46:0/2405742586 wait complete. 2023-12-16T21:14:54.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:14:55.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.002+0000 7fbca2c4f640 1 Processor -- start 2023-12-16T21:14:55.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.002+0000 7fbca2c4f640 1 -- start start 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbca2c4f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbc9c143490 0x7fbc9c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbca2c4f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 0x7fbc9c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbca2c4f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c15e2a0 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbca2c4f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c15b520 con 0x7fbc9c143490 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbca2c4f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c143870 con 0x7fbc9c15e730 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 0x7fbc9c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbca09eb640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbc9c143490 0x7fbc9c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 0x7fbc9c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:44138/0 (socket says 172.21.15.46:44138) 2023-12-16T21:14:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.003+0000 7fbc9bfff640 1 -- 172.21.15.46:0/3823366879 learned_addr learned my addr 172.21.15.46:0/3823366879 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.004+0000 7fbc9bfff640 1 -- 172.21.15.46:0/3823366879 >> v2:172.21.15.121:3300/0 conn(0x7fbc9c143490 msgr2=0x7fbc9c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.004+0000 7fbc9bfff640 1 --2- 172.21.15.46:0/3823366879 >> v2:172.21.15.121:3300/0 conn(0x7fbc9c143490 0x7fbc9c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.004+0000 7fbc9bfff640 1 -- 172.21.15.46:0/3823366879 >> v1:172.21.15.121:6789/0 conn(0x7fbc9c15a4e0 legacy=0x7fbc9c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.004+0000 7fbc9bfff640 1 -- 172.21.15.46:0/3823366879 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c15b240 con 0x7fbc9c15e730 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.004+0000 7fbc9bfff640 1 --2- 172.21.15.46:0/3823366879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 0x7fbc9c0b1230 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc8802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a48fd6aeac720fee server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbc9b7fe640 1 -- 172.21.15.46:0/3823366879 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88030840 con 0x7fbc9c15e730 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbc9b7fe640 1 -- 172.21.15.46:0/3823366879 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc880309a0 con 0x7fbc9c15e730 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbc9b7fe640 1 -- 172.21.15.46:0/3823366879 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88030d30 con 0x7fbc9c15e730 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 -- 172.21.15.46:0/3823366879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 msgr2=0x7fbc9c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/3823366879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 0x7fbc9c0b1230 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc8802b980 comp rx=0 tx=0).stop 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 -- 172.21.15.46:0/3823366879 shutdown_connections 2023-12-16T21:14:55.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/3823366879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c15e730 0x7fbc9c0b1230 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/3823366879 >> v2:172.21.15.121:3300/0 conn(0x7fbc9c143490 0x7fbc9c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 -- 172.21.15.46:0/3823366879 >> 172.21.15.46:0/3823366879 conn(0x7fbc9c06a4d0 msgr2=0x7fbc9c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 -- 172.21.15.46:0/3823366879 shutdown_connections 2023-12-16T21:14:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.005+0000 7fbca2c4f640 1 -- 172.21.15.46:0/3823366879 wait complete. 2023-12-16T21:14:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.006+0000 7fbca2c4f640 1 Processor -- start 2023-12-16T21:14:55.008 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.006+0000 7fbca2c4f640 1 -- start start 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca2c4f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c143490 0x7fbc9c13e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca2c4f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 0x7fbc9c0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca2c4f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc9c15e730 0x7fbc9c158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca2c4f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c142eb0 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca2c4f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c1422a0 con 0x7fbc9c15e730 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca2c4f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c142580 con 0x7fbc9c143490 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 0x7fbc9c0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 0x7fbc9c0aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39440/0 (socket says 172.21.15.46:39440) 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbc9bfff640 1 -- 172.21.15.46:0/2501721984 learned_addr learned my addr 172.21.15.46:0/2501721984 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca11ec640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc9c15e730 0x7fbc9c158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.007+0000 7fbca09eb640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c143490 0x7fbc9c13e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc9bfff640 1 -- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c143490 msgr2=0x7fbc9c13e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc9bfff640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c143490 0x7fbc9c13e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc9bfff640 1 -- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc9c15e730 msgr2=0x7fbc9c158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc9bfff640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc9c15e730 0x7fbc9c158e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc9bfff640 1 -- 172.21.15.46:0/2501721984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc880094d0 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.013 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc9bfff640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 0x7fbc9c0aff90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88039a90 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc88039bf0 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c142860 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc880433f0 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.008+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c13ea40 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.009+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c069400 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.014 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.010+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbc88042390 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.010+0000 7fbc997fa640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbc8006ae40 0x7fbc8006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.010+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fbc880ab860 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.010+0000 7fbca09eb640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbc8006ae40 0x7fbc8006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.011+0000 7fbca09eb640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbc8006ae40 0x7fbc8006d300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0ce2c0 tx=0x7fbc90009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:55.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.014+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fbc88076830 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.189+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7fbc9c0b2820 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.190+0000 7fbc997fa640 1 -- 172.21.15.46:0/2501721984 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7fbc88048020 con 0x7fbc9c15a4e0 2023-12-16T21:14:55.190 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:14:55.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbc8006ae40 msgr2=0x7fbc8006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbc8006ae40 0x7fbc8006d300 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0ce2c0 tx=0x7fbc90009190 comp rx=0 tx=0).stop 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 msgr2=0x7fbc9c0aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 0x7fbc9c0aff90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc88004730 comp rx=0 tx=0).stop 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 shutdown_connections 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbc8006ae40 0x7fbc8006d300 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbc9c15a4e0 0x7fbc9c0aff90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc9c15e730 0x7fbc9c158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 --2- 172.21.15.46:0/2501721984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbc9c143490 0x7fbc9c13e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 >> 172.21.15.46:0/2501721984 conn(0x7fbc9c06a4d0 msgr2=0x7fbc9c15a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 shutdown_connections 2023-12-16T21:14:55.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.192+0000 7fbca2c4f640 1 -- 172.21.15.46:0/2501721984 wait complete. 2023-12-16T21:14:55.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:14:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.310+0000 7fb03127c640 1 Processor -- start 2023-12-16T21:14:55.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.310+0000 7fb03127c640 1 -- start start 2023-12-16T21:14:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb03127c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c15d270 0x7fb02c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb03127c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c0709c0 0x7fb02c0cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb03127c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb02c0cdd10 0x7fb02c0cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb03127c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c0ce1a0 con 0x7fb02c0709c0 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb03127c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02c0d0c40 con 0x7fb02c0cdd10 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb03127c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c0ced90 con 0x7fb02c15d270 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb02a7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c0709c0 0x7fb02c0cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb02a7fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fb02c0709c0 0x7fb02c0cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39450/0 (socket says 172.21.15.46:39450) 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.311+0000 7fb02a7fc640 1 -- 172.21.15.46:0/1356891298 learned_addr learned my addr 172.21.15.46:0/1356891298 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02affd640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c15d270 0x7fb02c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02b7fe640 1 --2- 172.21.15.46:0/1356891298 >> v2:172.21.15.121:3300/0 conn(0x7fb02c0cdd10 0x7fb02c0cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02a7fc640 1 -- 172.21.15.46:0/1356891298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c15d270 msgr2=0x7fb02c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02a7fc640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c15d270 0x7fb02c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02a7fc640 1 -- 172.21.15.46:0/1356891298 >> v2:172.21.15.121:3300/0 conn(0x7fb02c0cdd10 msgr2=0x7fb02c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02a7fc640 1 --2- 172.21.15.46:0/1356891298 >> v2:172.21.15.121:3300/0 conn(0x7fb02c0cdd10 0x7fb02c0cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02a7fc640 1 -- 172.21.15.46:0/1356891298 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c0cf070 con 0x7fb02c0709c0 2023-12-16T21:14:55.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02affd640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c15d270 0x7fb02c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:14:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.312+0000 7fb02a7fc640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0709c0 0x7fb02c0cd7d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0b2ba0 tx=0x7fb01402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657dc949b45e7c18 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb029ffb640 1 -- 172.21.15.46:0/1356891298 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0140317d0 con 0x7fb02c0709c0 2023-12-16T21:14:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb029ffb640 1 -- 172.21.15.46:0/1356891298 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb014031930 con 0x7fb02c0709c0 2023-12-16T21:14:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb029ffb640 1 -- 172.21.15.46:0/1356891298 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014031cc0 con 0x7fb02c0709c0 2023-12-16T21:14:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 -- 172.21.15.46:0/1356891298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0709c0 msgr2=0x7fb02c0cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0709c0 0x7fb02c0cd7d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fb02c0b2ba0 tx=0x7fb01402b870 comp rx=0 tx=0).stop 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 -- 172.21.15.46:0/1356891298 shutdown_connections 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 --2- 172.21.15.46:0/1356891298 >> v2:172.21.15.121:3300/0 conn(0x7fb02c0cdd10 0x7fb02c0cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c0709c0 0x7fb02c0cd7d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 --2- 172.21.15.46:0/1356891298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c15d270 0x7fb02c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 -- 172.21.15.46:0/1356891298 >> 172.21.15.46:0/1356891298 conn(0x7fb02c0b2c10 msgr2=0x7fb02c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 -- 172.21.15.46:0/1356891298 shutdown_connections 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.313+0000 7fb03127c640 1 -- 172.21.15.46:0/1356891298 wait complete. 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.314+0000 7fb03127c640 1 Processor -- start 2023-12-16T21:14:55.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 -- start start 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb02c0709c0 0x7fb02c148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c0cdd10 0x7fb02c15d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 0x7fb02c0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c15f450 con 0x7fb02c15d270 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c15e7f0 con 0x7fb02c0709c0 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb03127c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c15df50 con 0x7fb02c0cdd10 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb02b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 0x7fb02c0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb02b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 0x7fb02c0ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39452/0 (socket says 172.21.15.46:39452) 2023-12-16T21:14:55.318 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.315+0000 7fb02b7fe640 1 -- 172.21.15.46:0/1258549243 learned_addr learned my addr 172.21.15.46:0/1258549243 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02a7fc640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c0cdd10 0x7fb02c15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02affd640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb02c0709c0 0x7fb02c148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02b7fe640 1 -- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c0cdd10 msgr2=0x7fb02c15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02b7fe640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c0cdd10 0x7fb02c15d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02b7fe640 1 -- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb02c0709c0 msgr2=0x7fb02c148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02b7fe640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb02c0709c0 0x7fb02c148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02b7fe640 1 -- 172.21.15.46:0/1258549243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0140094d0 con 0x7fb02c15d270 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02a7fc640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c0cdd10 0x7fb02c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb02b7fe640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 0x7fb02c0ca2e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00cc30 tx=0x7fb01c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c00ce40 con 0x7fb02c15d270 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c003990 con 0x7fb02c15d270 2023-12-16T21:14:55.319 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c003d20 con 0x7fb02c15d270 2023-12-16T21:14:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c15e230 con 0x7fb02c15d270 2023-12-16T21:14:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.316+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c0c46f0 con 0x7fb02c15d270 2023-12-16T21:14:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.317+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c067fc0 con 0x7fb02c15d270 2023-12-16T21:14:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.318+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb01c003e80 con 0x7fb02c15d270 2023-12-16T21:14:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.318+0000 7fb0237fe640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb00c06ae70 0x7fb00c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:14:55.320 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.318+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fb01c089aa0 con 0x7fb02c15d270 2023-12-16T21:14:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.318+0000 7fb02affd640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb00c06ae70 0x7fb00c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:14:55.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.319+0000 7fb02affd640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb00c06ae70 0x7fb00c06d330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb02c070270 tx=0x7fb024008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:14:55.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.322+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb01c054a50 con 0x7fb02c15d270 2023-12-16T21:14:55.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.510+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb02c15f590 con 0x7fb02c15d270 2023-12-16T21:14:55.511 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.510+0000 7fb0237fe640 1 -- 172.21.15.46:0/1258549243 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb02c15f590 con 0x7fb02c15d270 2023-12-16T21:14:55.511 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:14:55.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb00c06ae70 msgr2=0x7fb00c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb00c06ae70 0x7fb00c06d330 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb02c070270 tx=0x7fb024008040 comp rx=0 tx=0).stop 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 msgr2=0x7fb02c0ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 0x7fb02c0ca2e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb01c00cc30 tx=0x7fb01c00a5c0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 shutdown_connections 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb00c06ae70 0x7fb00c06d330 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb02c15d270 0x7fb02c0ca2e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb02c0cdd10 0x7fb02c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 --2- 172.21.15.46:0/1258549243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb02c0709c0 0x7fb02c148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 >> 172.21.15.46:0/1258549243 conn(0x7fb02c0b2c10 msgr2=0x7fb02c06b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 shutdown_connections 2023-12-16T21:14:55.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:14:55.513+0000 7fb03127c640 1 -- 172.21.15.46:0/1258549243 wait complete. 2023-12-16T21:14:55.524 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:14:55.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:14:55.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:14:55.553 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[105369]: fuse finished with error 0 and tester_r 0 2023-12-16T21:14:55.555 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:14:57.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:14:57.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:15:01.657 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:01.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:01.657 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:15:01.677 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2023-12-16T21:15:01.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.842+0000 7f24943c9640 1 Processor -- start 2023-12-16T21:15:01.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.843+0000 7f24943c9640 1 -- start start 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f24943c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f248c0b2180 0x7f248c06f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f24943c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f248c06f7f0 0x7f248c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f24943c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c147150 con 0x7f248c0cf490 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f24943c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f248c15e570 con 0x7f248c0b2180 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f24943c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f248c1620c0 con 0x7f248c06f7f0 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492165640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f248c0b2180 0x7f248c06f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492966640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f248c0cf490 0x7f248c06a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58398/0 (socket says 172.21.15.46:58398) 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492966640 1 -- 172.21.15.46:0/545352099 learned_addr learned my addr 172.21.15.46:0/545352099 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2491964640 1 --2- 172.21.15.46:0/545352099 >> v2:172.21.15.121:3300/0 conn(0x7f248c06f7f0 0x7f248c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492165640 1 -- 172.21.15.46:0/545352099 >> v2:172.21.15.121:3300/0 conn(0x7f248c06f7f0 msgr2=0x7f248c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:01.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492165640 1 --2- 172.21.15.46:0/545352099 >> v2:172.21.15.121:3300/0 conn(0x7f248c06f7f0 0x7f248c0cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:01.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492165640 1 -- 172.21.15.46:0/545352099 >> v1:172.21.15.46:6789/0 conn(0x7f248c0cf490 legacy=0x7f248c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:01.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2492165640 1 -- 172.21.15.46:0/545352099 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c0cfd80 con 0x7f248c0b2180 2023-12-16T21:15:01.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2491163640 1 -- 172.21.15.46:0/545352099 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3006131218 0 0) 0x7f248c147150 con 0x7f248c0cf490 2023-12-16T21:15:01.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.844+0000 7f2491964640 1 --2- 172.21.15.46:0/545352099 >> v2:172.21.15.121:3300/0 conn(0x7f248c06f7f0 0x7f248c0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.845+0000 7f2492165640 1 --2- 172.21.15.46:0/545352099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c06f2b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f248c0cff70 tx=0x7f248002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47def3065ccfa20d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.845+0000 7f2491163640 1 -- 172.21.15.46:0/545352099 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480031830 con 0x7f248c0b2180 2023-12-16T21:15:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.845+0000 7f2491163640 1 -- 172.21.15.46:0/545352099 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2480031990 con 0x7f248c0b2180 2023-12-16T21:15:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.845+0000 7f2491163640 1 -- 172.21.15.46:0/545352099 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2480031d20 con 0x7f248c0b2180 2023-12-16T21:15:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.845+0000 7f24943c9640 1 -- 172.21.15.46:0/545352099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 msgr2=0x7f248c06f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:01.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.845+0000 7f24943c9640 1 --2- 172.21.15.46:0/545352099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c06f2b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f248c0cff70 tx=0x7f248002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.846+0000 7f24943c9640 1 -- 172.21.15.46:0/545352099 shutdown_connections 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.846+0000 7f24943c9640 1 --2- 172.21.15.46:0/545352099 >> v2:172.21.15.121:3300/0 conn(0x7f248c06f7f0 0x7f248c0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.846+0000 7f24943c9640 1 --2- 172.21.15.46:0/545352099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c06f2b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.846+0000 7f24943c9640 1 -- 172.21.15.46:0/545352099 >> 172.21.15.46:0/545352099 conn(0x7f248c06d690 msgr2=0x7f248c06da70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.846+0000 7f24943c9640 1 -- 172.21.15.46:0/545352099 shutdown_connections 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.846+0000 7f24943c9640 1 -- 172.21.15.46:0/545352099 wait complete. 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.847+0000 7f24943c9640 1 Processor -- start 2023-12-16T21:15:01.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.847+0000 7f24943c9640 1 -- start start 2023-12-16T21:15:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f248c06f7f0 0x7f248c0aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c0cf490 0x7f248c15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:01.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c06fca0 con 0x7f248c0b2180 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c0af360 con 0x7f248c0cf490 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c137400 con 0x7f248c06f7f0 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34996/0 (socket says 172.21.15.46:34996) 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 -- 172.21.15.46:0/1447800749 learned_addr learned my addr 172.21.15.46:0/1447800749 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2492165640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f248c06f7f0 0x7f248c0aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2492966640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c0cf490 0x7f248c15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 -- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f248c06f7f0 msgr2=0x7f248c0aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f248c06f7f0 0x7f248c0aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 -- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c0cf490 msgr2=0x7f248c15bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c0cf490 0x7f248c15bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 -- 172.21.15.46:0/1447800749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2480009400 con 0x7f248c0b2180 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2492165640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f248c06f7f0 0x7f248c0aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f2491964640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c136ec0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f248800c820 tx=0x7f248800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:01.851 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24880079d0 con 0x7f248c0b2180 2023-12-16T21:15:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488007b30 con 0x7f248c0b2180 2023-12-16T21:15:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.848+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c15c4a0 con 0x7f248c0b2180 2023-12-16T21:15:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.849+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2488007ea0 con 0x7f248c0b2180 2023-12-16T21:15:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.849+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c0ae060 con 0x7f248c0b2180 2023-12-16T21:15:01.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.850+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c15e970 con 0x7f248c0b2180 2023-12-16T21:15:01.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.850+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f24880113f0 con 0x7f248c0b2180 2023-12-16T21:15:01.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.851+0000 7f247effd640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f247006ae40 0x7f247006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:01.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.851+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2488089800 con 0x7f248c0b2180 2023-12-16T21:15:01.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.851+0000 7f2492165640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f247006ae40 0x7f247006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:01.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.852+0000 7f2492165640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f247006ae40 0x7f247006d300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2480036000 tx=0x7f2480008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:01.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:01.855+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f248c15e970 con 0x7f248c0b2180 2023-12-16T21:15:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.102+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) v1 -- 0x7f248c15e970 con 0x7f248c0b2180 2023-12-16T21:15:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.104+0000 7f247effd640 1 -- 172.21.15.46:0/1447800749 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v17) v1 ==== 211+0+168 (secure 0 0 0) 0x7f248c15e970 con 0x7f248c0b2180 2023-12-16T21:15:02.104 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:15:02.105 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:15:02.105 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow rwp" 2023-12-16T21:15:02.105 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:15:02.105 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2023-12-16T21:15:02.105 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:15:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f247006ae40 msgr2=0x7f247006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f247006ae40 0x7f247006d300 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2480036000 tx=0x7f2480008030 comp rx=0 tx=0).stop 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 msgr2=0x7f248c136ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c136ec0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f248800c820 tx=0x7f248800cce0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 shutdown_connections 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f247006ae40 0x7f247006d300 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f248c0b2180 0x7f248c136ec0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f248c0cf490 0x7f248c15bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 --2- 172.21.15.46:0/1447800749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f248c06f7f0 0x7f248c0aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 >> 172.21.15.46:0/1447800749 conn(0x7f248c06d690 msgr2=0x7f248c0b33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 shutdown_connections 2023-12-16T21:15:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.106+0000 7f24943c9640 1 -- 172.21.15.46:0/1447800749 wait complete. 2023-12-16T21:15:02.117 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:15:02.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.223+0000 7f11ab72c640 1 Processor -- start 2023-12-16T21:15:02.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.223+0000 7f11ab72c640 1 -- start start 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.224+0000 7f11ab72c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.224+0000 7f11ab72c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11a415e730 0x7f11a40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.224+0000 7f11ab72c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11a415e2a0 con 0x7f11a415a4e0 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.224+0000 7f11ab72c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11a415b520 con 0x7f11a415e730 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.224+0000 7f11ab72c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a4143870 con 0x7f11a4143490 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a8cc7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11a415e730 0x7f11a40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:02.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a8cc7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11a415e730 0x7f11a40b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:60012/0 (socket says 172.21.15.46:60012) 2023-12-16T21:15:02.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a8cc7640 1 -- 172.21.15.46:0/3901351863 learned_addr learned my addr 172.21.15.46:0/3901351863 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:02.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a94c8640 1 --2- 172.21.15.46:0/3901351863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f119bfff640 1 -- 172.21.15.46:0/3901351863 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031426828 0 0) 0x7f11a415e2a0 con 0x7f11a415a4e0 2023-12-16T21:15:02.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f119bfff640 1 -- 172.21.15.46:0/3901351863 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1190003520 con 0x7f11a415a4e0 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a94c8640 1 -- 172.21.15.46:0/3901351863 >> v2:172.21.15.121:3300/0 conn(0x7f11a415e730 msgr2=0x7f11a40b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a94c8640 1 --2- 172.21.15.46:0/3901351863 >> v2:172.21.15.121:3300/0 conn(0x7f11a415e730 0x7f11a40b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a94c8640 1 -- 172.21.15.46:0/3901351863 >> v1:172.21.15.121:6789/0 conn(0x7f11a415a4e0 legacy=0x7f11a415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.225+0000 7f11a94c8640 1 -- 172.21.15.46:0/3901351863 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a415b240 con 0x7f11a4143490 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.226+0000 7f11a94c8640 1 --2- 172.21.15.46:0/3901351863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a40b0cf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f119c00bd90 tx=0x7f119c02dad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c1b6fc5aa220650 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.226+0000 7f119bfff640 1 -- 172.21.15.46:0/3901351863 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c02f890 con 0x7f11a4143490 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.226+0000 7f119bfff640 1 -- 172.21.15.46:0/3901351863 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c02f9f0 con 0x7f11a4143490 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.226+0000 7f119bfff640 1 -- 172.21.15.46:0/3901351863 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c02fd80 con 0x7f11a4143490 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 -- 172.21.15.46:0/3901351863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 msgr2=0x7f11a40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 --2- 172.21.15.46:0/3901351863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a40b0cf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f119c00bd90 tx=0x7f119c02dad0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 -- 172.21.15.46:0/3901351863 shutdown_connections 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 --2- 172.21.15.46:0/3901351863 >> v2:172.21.15.121:3300/0 conn(0x7f11a415e730 0x7f11a40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 --2- 172.21.15.46:0/3901351863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a40b0cf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 -- 172.21.15.46:0/3901351863 >> 172.21.15.46:0/3901351863 conn(0x7f11a406a4d0 msgr2=0x7f11a406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:02.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 -- 172.21.15.46:0/3901351863 shutdown_connections 2023-12-16T21:15:02.229 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.227+0000 7f11ab72c640 1 -- 172.21.15.46:0/3901351863 wait complete. 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.228+0000 7f11ab72c640 1 Processor -- start 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.228+0000 7f11ab72c640 1 -- start start 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11ab72c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a4159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11ab72c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11a415a4e0 0x7f11a40afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11ab72c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 0x7f11a40b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11ab72c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41e48e0 con 0x7f11a415e730 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11ab72c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41e4bc0 con 0x7f11a415a4e0 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11ab72c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11a41e4ea0 con 0x7f11a4143490 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 0x7f11a40b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 0x7f11a40b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35006/0 (socket says 172.21.15.46:35006) 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 -- 172.21.15.46:0/2371312225 learned_addr learned my addr 172.21.15.46:0/2371312225 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a94c8640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a4159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:02.231 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a8cc7640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11a415a4e0 0x7f11a40afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 -- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 msgr2=0x7f11a4159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a4159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 -- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11a415a4e0 msgr2=0x7f11a40afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11a415a4e0 0x7f11a40afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 -- 172.21.15.46:0/2371312225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c00ba70 con 0x7f11a415e730 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.229+0000 7f11a9cc9640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 0x7f11a40b0530 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f11a000c790 tx=0x7f11a000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.230+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000cea0 con 0x7f11a415e730 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.230+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a0003b70 con 0x7f11a415e730 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.230+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11a41e5180 con 0x7f11a415e730 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.230+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000a850 con 0x7f11a415e730 2023-12-16T21:15:02.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.230+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11a4159ea0 con 0x7f11a415e730 2023-12-16T21:15:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.231+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11a41e56a0 con 0x7f11a415e730 2023-12-16T21:15:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.231+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f11a000a9b0 con 0x7f11a415e730 2023-12-16T21:15:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.232+0000 7f1199ffb640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f118806ae40 0x7f118806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.232+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f11a0089500 con 0x7f11a415e730 2023-12-16T21:15:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.232+0000 7f11a94c8640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f118806ae40 0x7f118806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:02.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.232+0000 7f11a94c8640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f118806ae40 0x7f118806d300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f119c038000 tx=0x7f119c03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:02.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.236+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f11a41e56a0 con 0x7f11a415e730 2023-12-16T21:15:02.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.448+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f11a41e56a0 con 0x7f11a415e730 2023-12-16T21:15:02.449 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.448+0000 7f1199ffb640 1 -- 172.21.15.46:0/2371312225 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+82 (secure 0 0 0) 0x7f11a41e56a0 con 0x7f11a415e730 2023-12-16T21:15:02.449 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f118806ae40 msgr2=0x7f118806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f118806ae40 0x7f118806d300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f119c038000 tx=0x7f119c03b040 comp rx=0 tx=0).stop 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 msgr2=0x7f11a40b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 0x7f11a40b0530 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f11a000c790 tx=0x7f11a000cc50 comp rx=0 tx=0).stop 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 shutdown_connections 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f118806ae40 0x7f118806d300 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11a415e730 0x7f11a40b0530 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11a415a4e0 0x7f11a40afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 --2- 172.21.15.46:0/2371312225 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11a4143490 0x7f11a4159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 >> 172.21.15.46:0/2371312225 conn(0x7f11a406a4d0 msgr2=0x7f11a40ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:02.451 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 shutdown_connections 2023-12-16T21:15:02.452 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:02.450+0000 7f11ab72c640 1 -- 172.21.15.46:0/2371312225 wait complete. 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:15:02.461 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:15:02.462 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:15:02.462 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:02.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:02.462 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:15:02.477 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:02.478 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42582sec preferred_lft 42582sec 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:02.479 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::20ee:6cff:fee1:5615/64 scope link 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:02.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:02.480 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:15:02.548 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:15:02.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:02.550 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:15:02.606 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:15:02.607 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:15:02.607 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:15:02.607 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:15:02.607 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:15:02.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:02.607 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:15:02.607 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:15:02.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:15:02.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:15:02.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:15:02.608 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:15:02.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:02.714 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:15:02.729 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:15:02.730 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:15:02.730 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:02.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:02.730 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:15:02.801 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:15:02.801 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:02.801 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:02.856 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:02.857 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:02.857 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:02.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:02.984 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:03.040 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:03.041 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:03.042 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:03.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:03.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:03.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:03.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:03.043 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:03.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.044 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:03.097 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:15:03.097 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:15:03.140 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:03.170 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:03.227 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:03.227 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:03.227 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:03.227 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:03.228 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:03.229 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:03.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.230 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:03.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:15:03.233+0000 7febc9b87480 -1 init, newargv = 0x55d5165c6870 newargc=15 2023-12-16T21:15:03.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[106451]: starting ceph client 2023-12-16T21:15:03.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[106451]: starting fuse 2023-12-16T21:15:03.248 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:15:03.249 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:15:03.249 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:15:03.250 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:03.251 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:15:03.251 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:15:03.373 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.106451.asok 2023-12-16T21:15:03.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:15:03.384 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.106451.asok 2023-12-16T21:15:03.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.106451.asok status 2023-12-16T21:15:03.526 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:15:03.526 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "106451", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5661, 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:15:03.527 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 5661 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1650861349 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1650861349 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.5661 192.168.0.1:0/1650861349", 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1650861349", 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:15:03.528 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 14, 2023-12-16T21:15:03.529 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 37, 2023-12-16T21:15:03.529 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:15:03.529 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:15:03.529 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:15:03.529 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:15:03.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.535 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:15:03.535 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:03.551 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:15:03.551 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:03.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.551 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:03.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.622 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2023-12-16T21:15:03.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:03.680 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2023-12-16T21:15:03.739 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:03.740 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:03.795 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:03.941 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.941+0000 7f11d6ccf640 1 Processor -- start 2023-12-16T21:15:03.942 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.941+0000 7f11d6ccf640 1 -- start start 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d6ccf640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11d0144980 0x7f11d015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d6ccf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d00b4370 0x7f11d00b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d6ccf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d0071770 con 0x7f11d00ceb70 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d6ccf640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d00ccb60 con 0x7f11d0144980 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d6ccf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00b4ce0 con 0x7f11d00b4370 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11cffff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f11d00ceb70 0x7f11d00cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58412/0 (socket says 172.21.15.46:58412) 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11cffff640 1 -- 172.21.15.46:0/2676104550 learned_addr learned my addr 172.21.15.46:0/2676104550 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d526c640 1 --2- 172.21.15.46:0/2676104550 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d00b4370 0x7f11d00b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:03.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d4a6b640 1 --2- 172.21.15.46:0/2676104550 >> v2:172.21.15.121:3300/0 conn(0x7f11d0144980 0x7f11d015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:03.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11cf7fe640 1 -- 172.21.15.46:0/2676104550 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713567431 0 0) 0x7f11d0071770 con 0x7f11d00ceb70 2023-12-16T21:15:03.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11cf7fe640 1 -- 172.21.15.46:0/2676104550 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc003520 con 0x7f11d00ceb70 2023-12-16T21:15:03.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d526c640 1 -- 172.21.15.46:0/2676104550 >> v2:172.21.15.121:3300/0 conn(0x7f11d0144980 msgr2=0x7f11d015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:03.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d526c640 1 --2- 172.21.15.46:0/2676104550 >> v2:172.21.15.121:3300/0 conn(0x7f11d0144980 0x7f11d015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:03.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d526c640 1 -- 172.21.15.46:0/2676104550 >> v1:172.21.15.46:6789/0 conn(0x7f11d00ceb70 legacy=0x7f11d00cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:03.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.942+0000 7f11d526c640 1 -- 172.21.15.46:0/2676104550 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d00ccdc0 con 0x7f11d00b4370 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.943+0000 7f11d526c640 1 --2- 172.21.15.46:0/2676104550 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d00b4370 0x7f11d00b47a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009d90 tx=0x7f11c802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=655ffb96eb7eaee1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.943+0000 7f11cf7fe640 1 -- 172.21.15.46:0/2676104550 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c802cc50 con 0x7f11d00b4370 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.943+0000 7f11cf7fe640 1 -- 172.21.15.46:0/2676104550 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c802cdb0 con 0x7f11d00b4370 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.943+0000 7f11cf7fe640 1 -- 172.21.15.46:0/2676104550 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8034660 con 0x7f11d00b4370 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 -- 172.21.15.46:0/2676104550 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d00b4370 msgr2=0x7f11d00b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/2676104550 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d00b4370 0x7f11d00b47a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009d90 tx=0x7f11c802bea0 comp rx=0 tx=0).stop 2023-12-16T21:15:03.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 -- 172.21.15.46:0/2676104550 shutdown_connections 2023-12-16T21:15:03.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/2676104550 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d00b4370 0x7f11d00b47a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:03.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/2676104550 >> v2:172.21.15.121:3300/0 conn(0x7f11d0144980 0x7f11d015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:03.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 -- 172.21.15.46:0/2676104550 >> 172.21.15.46:0/2676104550 conn(0x7f11d006a4d0 msgr2=0x7f11d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:03.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 -- 172.21.15.46:0/2676104550 shutdown_connections 2023-12-16T21:15:03.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.944+0000 7f11d6ccf640 1 -- 172.21.15.46:0/2676104550 wait complete. 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.945+0000 7f11d6ccf640 1 Processor -- start 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.945+0000 7f11d6ccf640 1 -- start start 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d6ccf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d00b4370 0x7f11d01f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d6ccf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 0x7f11d015a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d6ccf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d0144980 0x7f11d015a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d6ccf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d01f0bd0 con 0x7f11d00ceb70 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d6ccf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d0158070 con 0x7f11d00b4370 2023-12-16T21:15:03.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d6ccf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d00b0150 con 0x7f11d0144980 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 0x7f11d015a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 0x7f11d015a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35016/0 (socket says 172.21.15.46:35016) 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 -- 172.21.15.46:0/3094754047 learned_addr learned my addr 172.21.15.46:0/3094754047 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d526c640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d0144980 0x7f11d015a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11d4a6b640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d00b4370 0x7f11d01f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 -- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d0144980 msgr2=0x7f11d015a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d0144980 0x7f11d015a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 -- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d00b4370 msgr2=0x7f11d01f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.946+0000 7f11cffff640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d00b4370 0x7f11d01f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11cffff640 1 -- 172.21.15.46:0/3094754047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c8009a70 con 0x7f11d00ceb70 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11cffff640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 0x7f11d015a230 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f11c000c530 tx=0x7f11c000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0005ce0 con 0x7f11d00ceb70 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c0005e40 con 0x7f11d00ceb70 2023-12-16T21:15:03.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d00af8b0 con 0x7f11d00ceb70 2023-12-16T21:15:03.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c00072b0 con 0x7f11d00ceb70 2023-12-16T21:15:03.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.947+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d00afe70 con 0x7f11d00ceb70 2023-12-16T21:15:03.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.948+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d0069400 con 0x7f11d00ceb70 2023-12-16T21:15:03.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.948+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f11c0007410 con 0x7f11d00ceb70 2023-12-16T21:15:03.950 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.949+0000 7f11cd7fa640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f11b006ae40 0x7f11b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:03.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.949+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f11c0088dc0 con 0x7f11d00ceb70 2023-12-16T21:15:03.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.949+0000 7f11d4a6b640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f11b006ae40 0x7f11b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:03.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.949+0000 7f11d4a6b640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f11b006ae40 0x7f11b006d300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009b40 tx=0x7f11b8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:03.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:03.953+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f11c0053dd0 con 0x7f11d00ceb70 2023-12-16T21:15:04.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.130+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11d015c960 con 0x7f11d00ceb70 2023-12-16T21:15:04.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.131+0000 7f11cd7fa640 1 -- 172.21.15.46:0/3094754047 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7f11c00573f0 con 0x7f11d00ceb70 2023-12-16T21:15:04.132 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:04.132 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f11b006ae40 msgr2=0x7f11b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f11b006ae40 0x7f11b006d300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009b40 tx=0x7f11b8009190 comp rx=0 tx=0).stop 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 msgr2=0x7f11d015a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 0x7f11d015a230 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f11c000c530 tx=0x7f11c000c9f0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 shutdown_connections 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f11b006ae40 0x7f11b006d300 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11d00ceb70 0x7f11d015a230 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11d0144980 0x7f11d015a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 --2- 172.21.15.46:0/3094754047 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11d00b4370 0x7f11d01f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 >> 172.21.15.46:0/3094754047 conn(0x7f11d006a4d0 msgr2=0x7f11d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:04.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 shutdown_connections 2023-12-16T21:15:04.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.133+0000 7f11d6ccf640 1 -- 172.21.15.46:0/3094754047 wait complete. 2023-12-16T21:15:04.145 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:04.145 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.255+0000 7f1f60a09640 1 Processor -- start 2023-12-16T21:15:04.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.255+0000 7f1f60a09640 1 -- start start 2023-12-16T21:15:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f60a09640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f5c160f70 0x7f1f5c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f60a09640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f60a09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5c1614e0 con 0x7f1f5c15d2d0 2023-12-16T21:15:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f60a09640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f5c1617c0 con 0x7f1f5c160f70 2023-12-16T21:15:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f60a09640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c161aa0 con 0x7f1f5c0ce480 2023-12-16T21:15:04.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f59d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f5c160f70 0x7f1f5c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f59d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1f5c160f70 0x7f1f5c0cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35018/0 (socket says 172.21.15.46:35018) 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f5a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1f5c15d2d0 0x7f1f5c0b8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58416/0 (socket says 172.21.15.46:58416) 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f59d9b640 1 -- 172.21.15.46:0/2612410313 learned_addr learned my addr 172.21.15.46:0/2612410313 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.256+0000 7f1f5ad9d640 1 --2- 172.21.15.46:0/2612410313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f59d9b640 1 -- 172.21.15.46:0/2612410313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 msgr2=0x7f1f5c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f59d9b640 1 --2- 172.21.15.46:0/2612410313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f59d9b640 1 -- 172.21.15.46:0/2612410313 >> v1:172.21.15.46:6789/0 conn(0x7f1f5c15d2d0 legacy=0x7f1f5c0b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f59d9b640 1 -- 172.21.15.46:0/2612410313 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c161d80 con 0x7f1f5c160f70 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f59d9b640 1 --2- 172.21.15.46:0/2612410313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c0cdf40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009870 tx=0x7f1f4402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f9fd18a8ca7df3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f5959a640 1 -- 172.21.15.46:0/2612410313 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44030840 con 0x7f1f5c160f70 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f5959a640 1 -- 172.21.15.46:0/2612410313 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f440309a0 con 0x7f1f5c160f70 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.257+0000 7f1f5959a640 1 -- 172.21.15.46:0/2612410313 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44030d30 con 0x7f1f5c160f70 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 -- 172.21.15.46:0/2612410313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 msgr2=0x7f1f5c0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2612410313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c0cdf40 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f1f44009870 tx=0x7f1f4402b980 comp rx=0 tx=0).stop 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 -- 172.21.15.46:0/2612410313 shutdown_connections 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2612410313 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2612410313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c0cdf40 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 -- 172.21.15.46:0/2612410313 >> 172.21.15.46:0/2612410313 conn(0x7f1f5c072230 msgr2=0x7f1f5c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 -- 172.21.15.46:0/2612410313 shutdown_connections 2023-12-16T21:15:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.258+0000 7f1f60a09640 1 -- 172.21.15.46:0/2612410313 wait complete. 2023-12-16T21:15:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.259+0000 7f1f60a09640 1 Processor -- start 2023-12-16T21:15:04.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 -- start start 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c15d2d0 0x7f1f5c0c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0c9090 con 0x7f1f5c160f70 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0c9370 con 0x7f1f5c15d2d0 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f60a09640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0c9650 con 0x7f1f5c0ce480 2023-12-16T21:15:04.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f5ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f5ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c157340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35028/0 (socket says 172.21.15.46:35028) 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.260+0000 7f1f5ad9d640 1 -- 172.21.15.46:0/2561365454 learned_addr learned my addr 172.21.15.46:0/2561365454 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5a59c640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f59d9b640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c15d2d0 0x7f1f5c0c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5ad9d640 1 -- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 msgr2=0x7f1f5c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5ad9d640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5ad9d640 1 -- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c15d2d0 msgr2=0x7f1f5c0c86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5ad9d640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c15d2d0 0x7f1f5c0c86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5ad9d640 1 -- 172.21.15.46:0/2561365454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f440094d0 con 0x7f1f5c160f70 2023-12-16T21:15:04.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f5ad9d640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c157340 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007e40 tx=0x7f1f4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c014620 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f4c00ee90 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5c0b3590 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c0149e0 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.261+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5c0ccaa0 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.262+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5c0b3ab0 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.263+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1f4c014b40 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.263+0000 7f1f52ffd640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f3c06ae40 0x7f1f3c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.263+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f1f4c08a1d0 con 0x7f1f5c160f70 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.263+0000 7f1f5a59c640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f3c06ae40 0x7f1f3c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.264+0000 7f1f5a59c640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f3c06ae40 0x7f1f3c06d300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f54000f60 tx=0x7f1f54007c80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.267+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1f5c0b3ab0 con 0x7f1f5c160f70 2023-12-16T21:15:04.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.445+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f5c0b3ab0 con 0x7f1f5c160f70 2023-12-16T21:15:04.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.445+0000 7f1f52ffd640 1 -- 172.21.15.46:0/2561365454 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f5c0b3ab0 con 0x7f1f5c160f70 2023-12-16T21:15:04.446 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f3c06ae40 msgr2=0x7f1f3c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f3c06ae40 0x7f1f3c06d300 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f1f54000f60 tx=0x7f1f54007c80 comp rx=0 tx=0).stop 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 msgr2=0x7f1f5c157340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c157340 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007e40 tx=0x7f1f4c00e4b0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 shutdown_connections 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1f3c06ae40 0x7f1f3c06d300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1f5c160f70 0x7f1f5c157340 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f5c15d2d0 0x7f1f5c0c86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 --2- 172.21.15.46:0/2561365454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f5c0ce480 0x7f1f5c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 >> 172.21.15.46:0/2561365454 conn(0x7f1f5c072230 msgr2=0x7f1f5c1556c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 shutdown_connections 2023-12-16T21:15:04.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.447+0000 7f1f60a09640 1 -- 172.21.15.46:0/2561365454 wait complete. 2023-12-16T21:15:04.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 Processor -- start 2023-12-16T21:15:04.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 -- start start 2023-12-16T21:15:04.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc140310 0x7f88bc1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f88bc1409d0 0x7f88bc06a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc1406f0 con 0x7f88bc1428d0 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88bc161e20 con 0x7f88bc1409d0 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.567+0000 7f88c287c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc143960 con 0x7f88bc140310 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88c0e19640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f88bc1428d0 0x7f88bc0714a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58424/0 (socket says 172.21.15.46:58424) 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88c0e19640 1 -- 172.21.15.46:0/1852449398 learned_addr learned my addr 172.21.15.46:0/1852449398 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88bbfff640 1 --2- 172.21.15.46:0/1852449398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc140310 0x7f88bc1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88bb7fe640 1 --2- 172.21.15.46:0/1852449398 >> v2:172.21.15.121:3300/0 conn(0x7f88bc1409d0 0x7f88bc06a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88baffd640 1 -- 172.21.15.46:0/1852449398 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412196858 0 0) 0x7f88bc1406f0 con 0x7f88bc1428d0 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88baffd640 1 -- 172.21.15.46:0/1852449398 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a4003520 con 0x7f88bc1428d0 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88bbfff640 1 -- 172.21.15.46:0/1852449398 >> v2:172.21.15.121:3300/0 conn(0x7f88bc1409d0 msgr2=0x7f88bc06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88bbfff640 1 --2- 172.21.15.46:0/1852449398 >> v2:172.21.15.121:3300/0 conn(0x7f88bc1409d0 0x7f88bc06a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88bbfff640 1 -- 172.21.15.46:0/1852449398 >> v1:172.21.15.46:6789/0 conn(0x7f88bc1428d0 legacy=0x7f88bc0714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.568+0000 7f88bbfff640 1 -- 172.21.15.46:0/1852449398 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc142de0 con 0x7f88bc140310 2023-12-16T21:15:04.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.569+0000 7f88bbfff640 1 --2- 172.21.15.46:0/1852449398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc140310 0x7f88bc1618e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0097a0 tx=0x7f88ac02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b40f0e35c0bd2ef server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.569+0000 7f88baffd640 1 -- 172.21.15.46:0/1852449398 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac02e070 con 0x7f88bc140310 2023-12-16T21:15:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.569+0000 7f88baffd640 1 -- 172.21.15.46:0/1852449398 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88ac0315a0 con 0x7f88bc140310 2023-12-16T21:15:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.569+0000 7f88baffd640 1 -- 172.21.15.46:0/1852449398 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88ac031930 con 0x7f88bc140310 2023-12-16T21:15:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 -- 172.21.15.46:0/1852449398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc140310 msgr2=0x7f88bc1618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 --2- 172.21.15.46:0/1852449398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc140310 0x7f88bc1618e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f88ac0097a0 tx=0x7f88ac02b840 comp rx=0 tx=0).stop 2023-12-16T21:15:04.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 -- 172.21.15.46:0/1852449398 shutdown_connections 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 --2- 172.21.15.46:0/1852449398 >> v2:172.21.15.121:3300/0 conn(0x7f88bc1409d0 0x7f88bc06a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 --2- 172.21.15.46:0/1852449398 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc140310 0x7f88bc1618e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 -- 172.21.15.46:0/1852449398 >> 172.21.15.46:0/1852449398 conn(0x7f88bc0b1e50 msgr2=0x7f88bc0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 -- 172.21.15.46:0/1852449398 shutdown_connections 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.570+0000 7f88c287c640 1 -- 172.21.15.46:0/1852449398 wait complete. 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.571+0000 7f88c287c640 1 Processor -- start 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.571+0000 7f88c287c640 1 -- start start 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c287c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88bc140310 0x7f88bc13d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c287c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc1409d0 0x7f88bc146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c287c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 0x7f88bc0aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c287c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc13df10 con 0x7f88bc1428d0 2023-12-16T21:15:04.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c287c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc144ff0 con 0x7f88bc140310 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c287c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc1452d0 con 0x7f88bc1409d0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 0x7f88bc0aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 0x7f88bc0aee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35030/0 (socket says 172.21.15.46:35030) 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 -- 172.21.15.46:0/3633871785 learned_addr learned my addr 172.21.15.46:0/3633871785 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88bb7fe640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc1409d0 0x7f88bc146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88bbfff640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88bc140310 0x7f88bc13d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 -- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc1409d0 msgr2=0x7f88bc146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc1409d0 0x7f88bc146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 -- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88bc140310 msgr2=0x7f88bc13d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88bc140310 0x7f88bc13d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 -- 172.21.15.46:0/3633871785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac009400 con 0x7f88bc1428d0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.572+0000 7f88c0e19640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 0x7f88bc0aee70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f88b400cb40 tx=0x7f88b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.573+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b400aca0 con 0x7f88bc1428d0 2023-12-16T21:15:04.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.573+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b400ae00 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.573+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc1455b0 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.573+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4012620 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.573+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc145b70 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.574+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc145d60 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.574+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f88b4007070 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.575+0000 7f88b8ff9640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.575+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f88b40896e0 con 0x7f88bc1428d0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.575+0000 7f88bbfff640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.575+0000 7f88bbfff640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88a806ae40 0x7f88a806d300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f88ac036000 tx=0x7f88ac03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.579+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f88b4054740 con 0x7f88bc1428d0 2023-12-16T21:15:04.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.754+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f88bc069400 con 0x7f88bc1428d0 2023-12-16T21:15:04.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.754+0000 7f88b8ff9640 1 -- 172.21.15.46:0/3633871785 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88b4057d60 con 0x7f88bc1428d0 2023-12-16T21:15:04.755 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:04.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88a806ae40 msgr2=0x7f88a806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88a806ae40 0x7f88a806d300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f88ac036000 tx=0x7f88ac03d040 comp rx=0 tx=0).stop 2023-12-16T21:15:04.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 msgr2=0x7f88bc0aee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 0x7f88bc0aee70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f88b400cb40 tx=0x7f88b400a060 comp rx=0 tx=0).stop 2023-12-16T21:15:04.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 shutdown_connections 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f88a806ae40 0x7f88a806d300 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88bc1428d0 0x7f88bc0aee70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88bc1409d0 0x7f88bc146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 --2- 172.21.15.46:0/3633871785 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88bc140310 0x7f88bc13d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 >> 172.21.15.46:0/3633871785 conn(0x7f88bc0b1e50 msgr2=0x7f88bc161670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 shutdown_connections 2023-12-16T21:15:04.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.756+0000 7f88c287c640 1 -- 172.21.15.46:0/3633871785 wait complete. 2023-12-16T21:15:04.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:04.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.877+0000 7f706c009640 1 Processor -- start 2023-12-16T21:15:04.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.877+0000 7f706c009640 1 -- start start 2023-12-16T21:15:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706c009640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f706415d2d0 0x7f70640b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706c009640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f70640cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706c009640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70640ce480 0x7f706406b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706c009640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70641614e0 con 0x7f706415d2d0 2023-12-16T21:15:04.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706c009640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70641617c0 con 0x7f70640ce480 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706c009640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7064161aa0 con 0x7f7064160f70 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f7069da5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f706415d2d0 0x7f70640b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f7069da5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f706415d2d0 0x7f70640b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35042/0 (socket says 172.21.15.46:35042) 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f7069da5640 1 -- 172.21.15.46:0/590520662 learned_addr learned my addr 172.21.15.46:0/590520662 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f70695a4640 1 --2- 172.21.15.46:0/590520662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f70640cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.878+0000 7f706a5a6640 1 --2- 172.21.15.46:0/590520662 >> v2:172.21.15.121:3300/0 conn(0x7f70640ce480 0x7f706406b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7069da5640 1 -- 172.21.15.46:0/590520662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 msgr2=0x7f70640cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7069da5640 1 --2- 172.21.15.46:0/590520662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f70640cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7069da5640 1 -- 172.21.15.46:0/590520662 >> v2:172.21.15.121:3300/0 conn(0x7f70640ce480 msgr2=0x7f706406b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:04.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7069da5640 1 --2- 172.21.15.46:0/590520662 >> v2:172.21.15.121:3300/0 conn(0x7f70640ce480 0x7f706406b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7069da5640 1 -- 172.21.15.46:0/590520662 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064161d80 con 0x7f706415d2d0 2023-12-16T21:15:04.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f706a5a6640 1 --2- 172.21.15.46:0/590520662 >> v2:172.21.15.121:3300/0 conn(0x7f70640ce480 0x7f706406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7069da5640 1 --2- 172.21.15.46:0/590520662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f706415d2d0 0x7f70640b8b90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f70640cdb80 tx=0x7f705402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a602e03d75bc30d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.879+0000 7f7068da3640 1 -- 172.21.15.46:0/590520662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705402ca70 con 0x7f706415d2d0 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f7068da3640 1 -- 172.21.15.46:0/590520662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705402cbd0 con 0x7f706415d2d0 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f7068da3640 1 -- 172.21.15.46:0/590520662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70540343f0 con 0x7f706415d2d0 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 -- 172.21.15.46:0/590520662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f706415d2d0 msgr2=0x7f70640b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 --2- 172.21.15.46:0/590520662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f706415d2d0 0x7f70640b8b90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f70640cdb80 tx=0x7f705402bed0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 -- 172.21.15.46:0/590520662 shutdown_connections 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 --2- 172.21.15.46:0/590520662 >> v2:172.21.15.121:3300/0 conn(0x7f70640ce480 0x7f706406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 --2- 172.21.15.46:0/590520662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f70640cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 --2- 172.21.15.46:0/590520662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f706415d2d0 0x7f70640b8b90 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 -- 172.21.15.46:0/590520662 >> 172.21.15.46:0/590520662 conn(0x7f7064072230 msgr2=0x7f70640cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 -- 172.21.15.46:0/590520662 shutdown_connections 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.880+0000 7f706c009640 1 -- 172.21.15.46:0/590520662 wait complete. 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.881+0000 7f706c009640 1 Processor -- start 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.881+0000 7f706c009640 1 -- start start 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706c009640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 0x7f7064158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706c009640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f706415d2d0 0x7f70640cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706c009640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f706415b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706c009640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706415ea20 con 0x7f70640ce480 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706c009640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706415dda0 con 0x7f706415d2d0 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706c009640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706415e080 con 0x7f7064160f70 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f7069da5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 0x7f7064158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f7069da5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 0x7f7064158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35058/0 (socket says 172.21.15.46:35058) 2023-12-16T21:15:04.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f7069da5640 1 -- 172.21.15.46:0/2781652126 learned_addr learned my addr 172.21.15.46:0/2781652126 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f706a5a6640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f706415b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.882+0000 7f70695a4640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f706415d2d0 0x7f70640cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f7069da5640 1 -- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 msgr2=0x7f706415b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f7069da5640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f706415b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f7069da5640 1 -- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f706415d2d0 msgr2=0x7f70640cc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f7069da5640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f706415d2d0 0x7f70640cc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f7069da5640 1 -- 172.21.15.46:0/2781652126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7054009b30 con 0x7f70640ce480 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f7069da5640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 0x7f7064158530 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7054000f80 tx=0x7f7054004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7054009510 con 0x7f70640ce480 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f705403e450 con 0x7f70640ce480 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f705403e800 con 0x7f70640ce480 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f706415e360 con 0x7f70640ce480 2023-12-16T21:15:04.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.883+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f706415da30 con 0x7f70640ce480 2023-12-16T21:15:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.884+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70640d0800 con 0x7f70640ce480 2023-12-16T21:15:04.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.884+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f705403e960 con 0x7f70640ce480 2023-12-16T21:15:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.885+0000 7f70627fc640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f705006ae40 0x7f705006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.885+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f70540ac500 con 0x7f70640ce480 2023-12-16T21:15:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.885+0000 7f70695a4640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f705006ae40 0x7f705006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:04.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.885+0000 7f70695a4640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f705006ae40 0x7f705006d300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7058006fd0 tx=0x7f7058008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:04.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:04.889+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f70540774b0 con 0x7f70640ce480 2023-12-16T21:15:05.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.064+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7064069400 con 0x7f70640ce480 2023-12-16T21:15:05.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.065+0000 7f70627fc640 1 -- 172.21.15.46:0/2781652126 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f705407aad0 con 0x7f70640ce480 2023-12-16T21:15:05.065 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f705006ae40 msgr2=0x7f705006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f705006ae40 0x7f705006d300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7058006fd0 tx=0x7f7058008040 comp rx=0 tx=0).stop 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 msgr2=0x7f7064158530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 0x7f7064158530 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7054000f80 tx=0x7f7054004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 shutdown_connections 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f705006ae40 0x7f705006d300 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7064160f70 0x7f706415b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f706415d2d0 0x7f70640cc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 --2- 172.21.15.46:0/2781652126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70640ce480 0x7f7064158530 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 >> 172.21.15.46:0/2781652126 conn(0x7f7064072230 msgr2=0x7f70640b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 shutdown_connections 2023-12-16T21:15:05.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.067+0000 7f706c009640 1 -- 172.21.15.46:0/2781652126 wait complete. 2023-12-16T21:15:05.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:15:05.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.186+0000 7f8c38501640 1 Processor -- start 2023-12-16T21:15:05.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.186+0000 7f8c38501640 1 -- start start 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.187+0000 7f8c38501640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 0x7f8c3006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.187+0000 7f8c38501640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8c3006bbf0 0x7f8c3006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.187+0000 7f8c38501640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c300adae0 con 0x7f8c3015d1d0 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.187+0000 7f8c38501640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c300addc0 con 0x7f8c3006bbf0 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.187+0000 7f8c38501640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300ae0a0 con 0x7f8c300ad640 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3629d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8c3015d1d0 0x7f8c3006b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58432/0 (socket says 172.21.15.46:58432) 2023-12-16T21:15:05.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3629d640 1 -- 172.21.15.46:0/3708079003 learned_addr learned my addr 172.21.15.46:0/3708079003 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c35a9c640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 0x7f8c3006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c36a9e640 1 --2- 172.21.15.46:0/3708079003 >> v2:172.21.15.121:3300/0 conn(0x7f8c3006bbf0 0x7f8c3006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19658621 0 0) 0x7f8c300adae0 con 0x7f8c3015d1d0 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c1c003520 con 0x7f8c3015d1d0 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2504224098 0 0) 0x7f8c1c003520 con 0x7f8c3015d1d0 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 msgr2=0x7f8c3006d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 0x7f8c3006d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 >> v2:172.21.15.121:3300/0 conn(0x7f8c3006bbf0 msgr2=0x7f8c3006d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 --2- 172.21.15.46:0/3708079003 >> v2:172.21.15.121:3300/0 conn(0x7f8c3006bbf0 0x7f8c3006d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300ae380 con 0x7f8c3015d1d0 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8c2c001670 con 0x7f8c3015d1d0 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 >> v1:172.21.15.46:6789/0 conn(0x7f8c3015d1d0 legacy=0x7f8c3006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c1c003970 0x7f8c1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 0x7f8c1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c004d00 0x7f8c1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300addc0 con 0x7f8c1c003970 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c300ae0a0 con 0x7f8c1c004d00 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c1c00b980 con 0x7f8c300ad640 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c3629d640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c1c003970 0x7f8c1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c35a9c640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 0x7f8c1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.188+0000 7f8c36a9e640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c004d00 0x7f8c1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3629d640 1 -- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 msgr2=0x7f8c1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3629d640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c300ad640 0x7f8c1c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3629d640 1 -- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c004d00 msgr2=0x7f8c1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3629d640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c004d00 0x7f8c1c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3629d640 1 -- 172.21.15.46:0/3708079003 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c003520 con 0x7f8c1c003970 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3629d640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c1c003970 0x7f8c1c003d90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00c240 tx=0x7f8c1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9ad48a7387b3f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.189+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c005ad0 con 0x7f8c1c003970 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c005c30 con 0x7f8c1c003970 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c3529b640 1 -- 172.21.15.46:0/3708079003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c02c8b0 con 0x7f8c1c003970 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c38501640 1 -- 172.21.15.46:0/3708079003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c1c003970 msgr2=0x7f8c1c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c38501640 1 --2- 172.21.15.46:0/3708079003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c1c003970 0x7f8c1c003d90 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00c240 tx=0x7f8c1c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c36a9e640 1 -- 172.21.15.46:0/3708079003 reap_dead start 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c38501640 1 -- 172.21.15.46:0/3708079003 shutdown_connections 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c38501640 1 -- 172.21.15.46:0/3708079003 >> 172.21.15.46:0/3708079003 conn(0x7f8c30070e70 msgr2=0x7f8c30071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c38501640 1 -- 172.21.15.46:0/3708079003 shutdown_connections 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.190+0000 7f8c38501640 1 -- 172.21.15.46:0/3708079003 wait complete. 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.191+0000 7f8c38501640 1 Processor -- start 2023-12-16T21:15:05.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.191+0000 7f8c38501640 1 -- start start 2023-12-16T21:15:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c38501640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c003970 0x7f8c301f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c38501640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c301f2640 0x7f8c301583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c38501640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 0x7f8c30158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c38501640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30158e60 con 0x7f8c30147fe0 2023-12-16T21:15:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c38501640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30142140 con 0x7f8c1c003970 2023-12-16T21:15:05.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c38501640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c30144db0 con 0x7f8c301f2640 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 0x7f8c30158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 0x7f8c30158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35076/0 (socket says 172.21.15.46:35076) 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 -- 172.21.15.46:0/1544446006 learned_addr learned my addr 172.21.15.46:0/1544446006 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c35a9c640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c301f2640 0x7f8c301583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c3629d640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c003970 0x7f8c301f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 -- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c301f2640 msgr2=0x7f8c301583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c301f2640 0x7f8c301583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 -- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c003970 msgr2=0x7f8c301f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.192+0000 7f8c36a9e640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c003970 0x7f8c301f2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c36a9e640 1 -- 172.21.15.46:0/1544446006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c30145090 con 0x7f8c30147fe0 2023-12-16T21:15:05.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c3629d640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c003970 0x7f8c301f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c36a9e640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 0x7f8c30158920 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8c30066840 tx=0x7f8c2800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800e040 con 0x7f8c30147fe0 2023-12-16T21:15:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2800f040 con 0x7f8c30147fe0 2023-12-16T21:15:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c30145370 con 0x7f8c30147fe0 2023-12-16T21:15:05.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c28015430 con 0x7f8c30147fe0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.193+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3015a760 con 0x7f8c30147fe0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.194+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3015a980 con 0x7f8c30147fe0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.194+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8c28015590 con 0x7f8c30147fe0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.195+0000 7f8c26ffd640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1406ae40 0x7f8c1406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.195+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8c28089f40 con 0x7f8c30147fe0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.195+0000 7f8c3629d640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1406ae40 0x7f8c1406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.195+0000 7f8c3629d640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1406ae40 0x7f8c1406d300 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c034000 tx=0x7f8c2c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.199+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8c28054dc0 con 0x7f8c30147fe0 2023-12-16T21:15:05.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.374+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f8c30069400 con 0x7f8c30147fe0 2023-12-16T21:15:05.375 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.374+0000 7f8c26ffd640 1 -- 172.21.15.46:0/1544446006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7f8c280583e0 con 0x7f8c30147fe0 2023-12-16T21:15:05.375 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1406ae40 msgr2=0x7f8c1406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1406ae40 0x7f8c1406d300 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c034000 tx=0x7f8c2c009030 comp rx=0 tx=0).stop 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 msgr2=0x7f8c30158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 0x7f8c30158920 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8c30066840 tx=0x7f8c2800f580 comp rx=0 tx=0).stop 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 shutdown_connections 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8c1406ae40 0x7f8c1406d300 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c30147fe0 0x7f8c30158920 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c301f2640 0x7f8c301583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 --2- 172.21.15.46:0/1544446006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c1c003970 0x7f8c301f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 >> 172.21.15.46:0/1544446006 conn(0x7f8c30070e70 msgr2=0x7f8c3006e830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 shutdown_connections 2023-12-16T21:15:05.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.376+0000 7f8c38501640 1 -- 172.21.15.46:0/1544446006 wait complete. 2023-12-16T21:15:05.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 Processor -- start 2023-12-16T21:15:05.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 -- start start 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fad6815d2e0 0x7fad680d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d6c0 0x7fad680d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad6806ce00 con 0x7fad6806b4c0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6806c1f0 con 0x7fad6815d2e0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.495+0000 7fad6d933640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6806c4d0 con 0x7fad6815d6c0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad677fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fad6806b4c0 0x7fad6806b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58448/0 (socket says 172.21.15.46:58448) 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad677fe640 1 -- 172.21.15.46:0/635375862 learned_addr learned my addr 172.21.15.46:0/635375862 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad66ffd640 1 --2- 172.21.15.46:0/635375862 >> v2:172.21.15.121:3300/0 conn(0x7fad6815d2e0 0x7fad680d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad667fc640 1 --2- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d6c0 0x7fad680d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad65ffb640 1 -- 172.21.15.46:0/635375862 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002546542 0 0) 0x7fad6806ce00 con 0x7fad6806b4c0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad65ffb640 1 -- 172.21.15.46:0/635375862 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad54003520 con 0x7fad6806b4c0 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad66ffd640 1 -- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d6c0 msgr2=0x7fad680d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad66ffd640 1 --2- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d6c0 0x7fad680d1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad66ffd640 1 -- 172.21.15.46:0/635375862 >> v1:172.21.15.46:6789/0 conn(0x7fad6806b4c0 legacy=0x7fad6806b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.496+0000 7fad66ffd640 1 -- 172.21.15.46:0/635375862 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6806c7b0 con 0x7fad6815d2e0 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.497+0000 7fad66ffd640 1 --2- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6815d2e0 0x7fad680d1640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fad58009d90 tx=0x7fad5802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a49fd0a5ddf9b664 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.497+0000 7fad65ffb640 1 -- 172.21.15.46:0/635375862 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5802cbc0 con 0x7fad6815d2e0 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.497+0000 7fad65ffb640 1 -- 172.21.15.46:0/635375862 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5802cd20 con 0x7fad6815d2e0 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.497+0000 7fad65ffb640 1 -- 172.21.15.46:0/635375862 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad58034640 con 0x7fad6815d2e0 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 -- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6815d2e0 msgr2=0x7fad680d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 --2- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6815d2e0 0x7fad680d1640 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fad58009d90 tx=0x7fad5802be10 comp rx=0 tx=0).stop 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 -- 172.21.15.46:0/635375862 shutdown_connections 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 --2- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d6c0 0x7fad680d1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 --2- 172.21.15.46:0/635375862 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6815d2e0 0x7fad680d1640 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 -- 172.21.15.46:0/635375862 >> 172.21.15.46:0/635375862 conn(0x7fad6809c580 msgr2=0x7fad680d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:05.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 -- 172.21.15.46:0/635375862 shutdown_connections 2023-12-16T21:15:05.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.498+0000 7fad6d933640 1 -- 172.21.15.46:0/635375862 wait complete. 2023-12-16T21:15:05.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.499+0000 7fad6d933640 1 Processor -- start 2023-12-16T21:15:05.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.499+0000 7fad6d933640 1 -- start start 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad6d933640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6806b4c0 0x7fad68159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad6d933640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d2e0 0x7fad6814c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad6d933640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 0x7fad680c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad6d933640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680b8630 con 0x7fad6815d6c0 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad6d933640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6814cb60 con 0x7fad6806b4c0 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad6d933640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad680b79b0 con 0x7fad6815d2e0 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 0x7fad680c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 0x7fad680c6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35090/0 (socket says 172.21.15.46:35090) 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 -- 172.21.15.46:0/211715651 learned_addr learned my addr 172.21.15.46:0/211715651 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad667fc640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d2e0 0x7fad6814c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad66ffd640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6806b4c0 0x7fad68159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 -- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d2e0 msgr2=0x7fad6814c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d2e0 0x7fad6814c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 -- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6806b4c0 msgr2=0x7fad68159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6806b4c0 0x7fad68159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.500+0000 7fad677fe640 1 -- 172.21.15.46:0/211715651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad58009a70 con 0x7fad6815d6c0 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.501+0000 7fad677fe640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 0x7fad680c6780 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fad6000c430 tx=0x7fad6000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.501+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad60005ce0 con 0x7fad6815d6c0 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.501+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad60005e40 con 0x7fad6815d6c0 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.501+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad680b7c90 con 0x7fad6815d6c0 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.501+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad60007310 con 0x7fad6815d6c0 2023-12-16T21:15:05.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.501+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad680b8250 con 0x7fad6815d6c0 2023-12-16T21:15:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.502+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad680b8770 con 0x7fad6815d6c0 2023-12-16T21:15:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.502+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fad60007470 con 0x7fad6815d6c0 2023-12-16T21:15:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.503+0000 7fad5ffff640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fad4806ae40 0x7fad4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.503+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fad60088db0 con 0x7fad6815d6c0 2023-12-16T21:15:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.503+0000 7fad66ffd640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fad4806ae40 0x7fad4806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.503+0000 7fad66ffd640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fad4806ae40 0x7fad4806d300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fad58036000 tx=0x7fad58030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.507+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fad680b8770 con 0x7fad6815d6c0 2023-12-16T21:15:05.696 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.696+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fad680b8770 con 0x7fad6815d6c0 2023-12-16T21:15:05.697 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.696+0000 7fad5ffff640 1 -- 172.21.15.46:0/211715651 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7fad680b8770 con 0x7fad6815d6c0 2023-12-16T21:15:05.697 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fad4806ae40 msgr2=0x7fad4806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fad4806ae40 0x7fad4806d300 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fad58036000 tx=0x7fad58030000 comp rx=0 tx=0).stop 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 msgr2=0x7fad680c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 0x7fad680c6780 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fad6000c430 tx=0x7fad6000c8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 shutdown_connections 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fad4806ae40 0x7fad4806d300 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fad6815d6c0 0x7fad680c6780 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fad6815d2e0 0x7fad6814c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 --2- 172.21.15.46:0/211715651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fad6806b4c0 0x7fad68159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 >> 172.21.15.46:0/211715651 conn(0x7fad6809c580 msgr2=0x7fad6815dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 shutdown_connections 2023-12-16T21:15:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.698+0000 7fad6d933640 1 -- 172.21.15.46:0/211715651 wait complete. 2023-12-16T21:15:05.709 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:05.709 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:05.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:05.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.871+0000 7f1d9befc640 1 Processor -- start 2023-12-16T21:15:05.871 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.871+0000 7f1d9befc640 1 -- start start 2023-12-16T21:15:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9befc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d940d3900 0x7f1d940b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.872 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9befc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d940b2b10 con 0x7f1d940d2910 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9befc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d940d1590 con 0x7f1d9415d330 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9befc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d940d1870 con 0x7f1d940d3900 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9a499640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d940d3900 0x7f1d940b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d99497640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f1d940d2910 0x7f1d940d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58460/0 (socket says 172.21.15.46:58460) 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d99497640 1 -- 172.21.15.46:0/1846180975 learned_addr learned my addr 172.21.15.46:0/1846180975 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9a499640 1 -- 172.21.15.46:0/1846180975 >> v1:172.21.15.121:6789/0 conn(0x7f1d9415d330 legacy=0x7f1d940b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9a499640 1 -- 172.21.15.46:0/1846180975 >> v1:172.21.15.46:6789/0 conn(0x7f1d940d2910 legacy=0x7f1d940d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d9a499640 1 -- 172.21.15.46:0/1846180975 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d940d1b50 con 0x7f1d940d3900 2023-12-16T21:15:05.873 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.872+0000 7f1d98c96640 1 -- 172.21.15.46:0/1846180975 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784902908 0 0) 0x7f1d940b2b10 con 0x7f1d940d2910 2023-12-16T21:15:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d9a499640 1 --2- 172.21.15.46:0/1846180975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d3900 0x7f1d940b6390 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1d840097a0 tx=0x7f1d8402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614348160b57c45f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d98c96640 1 -- 172.21.15.46:0/1846180975 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8402e070 con 0x7f1d940d3900 2023-12-16T21:15:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d98c96640 1 -- 172.21.15.46:0/1846180975 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d840315a0 con 0x7f1d940d3900 2023-12-16T21:15:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d98c96640 1 -- 172.21.15.46:0/1846180975 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84031930 con 0x7f1d940d3900 2023-12-16T21:15:05.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d9befc640 1 -- 172.21.15.46:0/1846180975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d3900 msgr2=0x7f1d940b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1846180975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d3900 0x7f1d940b6390 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1d840097a0 tx=0x7f1d8402b840 comp rx=0 tx=0).stop 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.873+0000 7f1d9befc640 1 -- 172.21.15.46:0/1846180975 shutdown_connections 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.874+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1846180975 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d3900 0x7f1d940b6390 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.874+0000 7f1d9befc640 1 -- 172.21.15.46:0/1846180975 >> 172.21.15.46:0/1846180975 conn(0x7f1d94072020 msgr2=0x7f1d94072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.874+0000 7f1d9befc640 1 -- 172.21.15.46:0/1846180975 shutdown_connections 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.874+0000 7f1d9befc640 1 -- 172.21.15.46:0/1846180975 wait complete. 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 Processor -- start 2023-12-16T21:15:05.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 -- start start 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 0x7f1d94159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d940d3900 0x7f1d940ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9415d330 0x7f1d940caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d941606b0 con 0x7f1d940d2910 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9415faa0 con 0x7f1d940d3900 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.875+0000 7f1d9befc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9415fd80 con 0x7f1d9415d330 2023-12-16T21:15:05.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 0x7f1d94159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 0x7f1d94159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35100/0 (socket says 172.21.15.46:35100) 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 -- 172.21.15.46:0/1861312287 learned_addr learned my addr 172.21.15.46:0/1861312287 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d9a499640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9415d330 0x7f1d940caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99497640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d940d3900 0x7f1d940ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 -- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9415d330 msgr2=0x7f1d940caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9415d330 0x7f1d940caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 -- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d940d3900 msgr2=0x7f1d940ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d940d3900 0x7f1d940ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 -- 172.21.15.46:0/1861312287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d84009400 con 0x7f1d940d2910 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d9a499640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9415d330 0x7f1d940caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d99c98640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 0x7f1d94159310 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00ba50 tx=0x7f1d8c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c00cc10 con 0x7f1d940d2910 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8c00cd70 con 0x7f1d940d2910 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c014680 con 0x7f1d940d2910 2023-12-16T21:15:05.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d94160060 con 0x7f1d940d2910 2023-12-16T21:15:05.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.876+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d94159850 con 0x7f1d940d2910 2023-12-16T21:15:05.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.877+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d94069400 con 0x7f1d940d2910 2023-12-16T21:15:05.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.878+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1d8c007370 con 0x7f1d940d2910 2023-12-16T21:15:05.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.878+0000 7f1d927fc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d8006ae70 0x7f1d8006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.878+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f1d8c089270 con 0x7f1d940d2910 2023-12-16T21:15:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.878+0000 7f1d99497640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d8006ae70 0x7f1d8006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:05.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.879+0000 7f1d99497640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d8006ae70 0x7f1d8006d330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d940c9db0 tx=0x7f1d88007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:05.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:05.882+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1d8c054240 con 0x7f1d940d2910 2023-12-16T21:15:06.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.063+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d940b66c0 con 0x7f1d940d2910 2023-12-16T21:15:06.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.064+0000 7f1d927fc640 1 -- 172.21.15.46:0/1861312287 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7f1d8c057860 con 0x7f1d940d2910 2023-12-16T21:15:06.065 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:06.065 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d8006ae70 msgr2=0x7f1d8006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d8006ae70 0x7f1d8006d330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1d940c9db0 tx=0x7f1d88007d50 comp rx=0 tx=0).stop 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 msgr2=0x7f1d94159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 0x7f1d94159310 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c00ba50 tx=0x7f1d8c00bf10 comp rx=0 tx=0).stop 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 shutdown_connections 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d8006ae70 0x7f1d8006d330 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9415d330 0x7f1d940caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d940d3900 0x7f1d940ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 --2- 172.21.15.46:0/1861312287 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d940d2910 0x7f1d94159310 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 >> 172.21.15.46:0/1861312287 conn(0x7f1d94072020 msgr2=0x7f1d940b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 shutdown_connections 2023-12-16T21:15:06.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.066+0000 7f1d9befc640 1 -- 172.21.15.46:0/1861312287 wait complete. 2023-12-16T21:15:06.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:06.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.187+0000 7fe0ffc1e640 1 Processor -- start 2023-12-16T21:15:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.187+0000 7fe0ffc1e640 1 -- start start 2023-12-16T21:15:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0ffc1e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f8143490 0x7fe0f80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0ffc1e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f815e2a0 con 0x7fe0f815e730 2023-12-16T21:15:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0ffc1e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f815b240 con 0x7fe0f815abf0 2023-12-16T21:15:06.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0ffc1e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f815b520 con 0x7fe0f8143490 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0fd1b9640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe0f815e730 0x7fe0f815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58462/0 (socket says 172.21.15.46:58462) 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0fd1b9640 1 -- 172.21.15.46:0/4283630850 learned_addr learned my addr 172.21.15.46:0/4283630850 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0fd9ba640 1 --2- 172.21.15.46:0/4283630850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f8143490 0x7fe0f80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.188+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118989993 0 0) 0x7fe0f815e2a0 con 0x7fe0f815e730 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.189+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e0003520 con 0x7fe0f815e730 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.189+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609799131 0 0) 0x7fe0f815b240 con 0x7fe0f815abf0 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.189+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f815e2a0 con 0x7fe0f815abf0 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.189+0000 7fe0fd9ba640 1 -- 172.21.15.46:0/4283630850 >> v1:172.21.15.121:6789/0 conn(0x7fe0f815abf0 legacy=0x7fe0f80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.189+0000 7fe0fd9ba640 1 -- 172.21.15.46:0/4283630850 >> v1:172.21.15.46:6789/0 conn(0x7fe0f815e730 legacy=0x7fe0f815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.189+0000 7fe0fd9ba640 1 -- 172.21.15.46:0/4283630850 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f815dc50 con 0x7fe0f8143490 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0fd9ba640 1 --2- 172.21.15.46:0/4283630850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f8143490 0x7fe0f80b0cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00097d0 tx=0x7fe0f002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e78b82d5d24ad1db server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0031890 con 0x7fe0f8143490 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f00319f0 con 0x7fe0f8143490 2023-12-16T21:15:06.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0fc9b8640 1 -- 172.21.15.46:0/4283630850 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0031d80 con 0x7fe0f8143490 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/4283630850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f8143490 msgr2=0x7fe0f80b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/4283630850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f8143490 0x7fe0f80b0cf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00097d0 tx=0x7fe0f002b930 comp rx=0 tx=0).stop 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/4283630850 shutdown_connections 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/4283630850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f8143490 0x7fe0f80b0cf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/4283630850 >> 172.21.15.46:0/4283630850 conn(0x7fe0f806a4d0 msgr2=0x7fe0f806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/4283630850 shutdown_connections 2023-12-16T21:15:06.193 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.190+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/4283630850 wait complete. 2023-12-16T21:15:06.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.191+0000 7fe0ffc1e640 1 Processor -- start 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 -- start start 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0f8143490 0x7fe0f8159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f815abf0 0x7fe0f80ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 0x7fe0f80ab6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81412a0 con 0x7fe0f815e730 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f8140fc0 con 0x7fe0f8143490 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0ffc1e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80a88d0 con 0x7fe0f815abf0 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0fe1bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 0x7fe0f80ab6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0fe1bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 0x7fe0f80ab6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35110/0 (socket says 172.21.15.46:35110) 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0fe1bb640 1 -- 172.21.15.46:0/621904392 learned_addr learned my addr 172.21.15.46:0/621904392 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0fd9ba640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0f8143490 0x7fe0f8159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.192+0000 7fe0fd1b9640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f815abf0 0x7fe0f80ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0fe1bb640 1 -- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f815abf0 msgr2=0x7fe0f80ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0fe1bb640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f815abf0 0x7fe0f80ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0fe1bb640 1 -- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0f8143490 msgr2=0x7fe0f8159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0fe1bb640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0f8143490 0x7fe0f8159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0fe1bb640 1 -- 172.21.15.46:0/621904392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f0009400 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0fe1bb640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 0x7fe0f80ab6d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800c790 tx=0x7fe0e800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e800cea0 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e8003b70 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f80a8bb0 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e800a850 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.193+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f8159a70 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.194+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f80a90d0 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.195+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe0e800a9b0 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.195+0000 7fe0f67fc640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe0e406ae70 0x7fe0e406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.195+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fe0e80894d0 con 0x7fe0f815e730 2023-12-16T21:15:06.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.195+0000 7fe0fd9ba640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe0e406ae70 0x7fe0e406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.196+0000 7fe0fd9ba640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe0e406ae70 0x7fe0e406d330 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0036000 tx=0x7fe0f003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.199+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe0f80a90d0 con 0x7fe0f815e730 2023-12-16T21:15:06.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.373+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0f80a90d0 con 0x7fe0f815e730 2023-12-16T21:15:06.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.374+0000 7fe0f67fc640 1 -- 172.21.15.46:0/621904392 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0f80a90d0 con 0x7fe0f815e730 2023-12-16T21:15:06.375 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:06.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe0e406ae70 msgr2=0x7fe0e406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe0e406ae70 0x7fe0e406d330 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0036000 tx=0x7fe0f003c040 comp rx=0 tx=0).stop 2023-12-16T21:15:06.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 msgr2=0x7fe0f80ab6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 0x7fe0f80ab6d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800c790 tx=0x7fe0e800cc50 comp rx=0 tx=0).stop 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 shutdown_connections 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe0e406ae70 0x7fe0e406d330 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0f815e730 0x7fe0f80ab6d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0f815abf0 0x7fe0f80ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 --2- 172.21.15.46:0/621904392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0f8143490 0x7fe0f8159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 >> 172.21.15.46:0/621904392 conn(0x7fe0f806a4d0 msgr2=0x7fe0f8070290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 shutdown_connections 2023-12-16T21:15:06.377 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.376+0000 7fe0ffc1e640 1 -- 172.21.15.46:0/621904392 wait complete. 2023-12-16T21:15:06.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.496+0000 7f2c22089640 1 Processor -- start 2023-12-16T21:15:06.497 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.496+0000 7f2c22089640 1 -- start start 2023-12-16T21:15:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c22089640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c1c15d330 0x7f2c1c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c22089640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2c1c0d3900 0x7f2c1c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c22089640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c0b2b10 con 0x7f2c1c0d2910 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c22089640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c0d1590 con 0x7f2c1c15d330 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c22089640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c0d1870 con 0x7f2c1c0d3900 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c1b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c1c15d330 0x7f2c1c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c1b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2c1c15d330 0x7f2c1c0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35114/0 (socket says 172.21.15.46:35114) 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c1b7fe640 1 -- 172.21.15.46:0/706532468 learned_addr learned my addr 172.21.15.46:0/706532468 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c1bfff640 1 --2- 172.21.15.46:0/706532468 >> v2:172.21.15.121:3300/0 conn(0x7f2c1c0d3900 0x7f2c1c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c1b7fe640 1 -- 172.21.15.46:0/706532468 >> v2:172.21.15.121:3300/0 conn(0x7f2c1c0d3900 msgr2=0x7f2c1c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.497+0000 7f2c1b7fe640 1 --2- 172.21.15.46:0/706532468 >> v2:172.21.15.121:3300/0 conn(0x7f2c1c0d3900 0x7f2c1c0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.498+0000 7f2c1b7fe640 1 -- 172.21.15.46:0/706532468 >> v1:172.21.15.121:6789/0 conn(0x7f2c1c0d2910 legacy=0x7f2c1c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.498+0000 7f2c1b7fe640 1 -- 172.21.15.46:0/706532468 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0d1b50 con 0x7f2c1c15d330 2023-12-16T21:15:06.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.498+0000 7f2c1b7fe640 1 --2- 172.21.15.46:0/706532468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0b8b80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c06c500 tx=0x7f2c0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af651b3d7052487 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.498+0000 7f2c1a7fc640 1 -- 172.21.15.46:0/706532468 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c031870 con 0x7f2c1c15d330 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.498+0000 7f2c1a7fc640 1 -- 172.21.15.46:0/706532468 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c0319d0 con 0x7f2c1c15d330 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.498+0000 7f2c1a7fc640 1 -- 172.21.15.46:0/706532468 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c031d60 con 0x7f2c1c15d330 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 -- 172.21.15.46:0/706532468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c15d330 msgr2=0x7f2c1c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 --2- 172.21.15.46:0/706532468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0b8b80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c06c500 tx=0x7f2c0c02b910 comp rx=0 tx=0).stop 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 -- 172.21.15.46:0/706532468 shutdown_connections 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 --2- 172.21.15.46:0/706532468 >> v2:172.21.15.121:3300/0 conn(0x7f2c1c0d3900 0x7f2c1c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 --2- 172.21.15.46:0/706532468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0b8b80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 -- 172.21.15.46:0/706532468 >> 172.21.15.46:0/706532468 conn(0x7f2c1c072020 msgr2=0x7f2c1c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 -- 172.21.15.46:0/706532468 shutdown_connections 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.499+0000 7f2c22089640 1 -- 172.21.15.46:0/706532468 wait complete. 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.500+0000 7f2c22089640 1 Processor -- start 2023-12-16T21:15:06.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.500+0000 7f2c22089640 1 -- start start 2023-12-16T21:15:06.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c22089640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1c0d2910 0x7f2c1c0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c22089640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 0x7f2c1c148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c22089640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c22089640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c15e550 con 0x7f2c1c0d3900 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c22089640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0c6fb0 con 0x7f2c1c15d330 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c22089640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0cb320 con 0x7f2c1c0d2910 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c1affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 0x7f2c1c148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c1affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 0x7f2c1c148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35130/0 (socket says 172.21.15.46:35130) 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.501+0000 7f2c1affd640 1 -- 172.21.15.46:0/2794998968 learned_addr learned my addr 172.21.15.46:0/2794998968 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1bfff640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1b7fe640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1c0d2910 0x7f2c1c0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1affd640 1 -- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1c0d2910 msgr2=0x7f2c1c0cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1affd640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1c0d2910 0x7f2c1c0cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.504 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1affd640 1 -- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c15d330 msgr2=0x7f2c1c0c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1affd640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1affd640 1 -- 172.21.15.46:0/2794998968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c0094d0 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c1affd640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 0x7f2c1c148750 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c400 tx=0x7f2c1000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100052d0 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c0cb600 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c10005430 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c10003830 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.502+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c0cb8e0 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.503+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c0c7140 con 0x7f2c1c0d3900 2023-12-16T21:15:06.505 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.503+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2c10003990 con 0x7f2c1c0d3900 2023-12-16T21:15:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.504+0000 7f2c03fff640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2c0806ae70 0x7f2c0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.504+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2c10088d90 con 0x7f2c1c0d3900 2023-12-16T21:15:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.504+0000 7f2c1b7fe640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2c0806ae70 0x7f2c0806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.504+0000 7f2c1b7fe640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2c0806ae70 0x7f2c0806d330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c036000 tx=0x7f2c0c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.508+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2c10053c30 con 0x7f2c1c0d3900 2023-12-16T21:15:06.685 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.685+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c1c069400 con 0x7f2c1c0d3900 2023-12-16T21:15:06.686 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.685+0000 7f2c03fff640 1 -- 172.21.15.46:0/2794998968 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c10057250 con 0x7f2c1c0d3900 2023-12-16T21:15:06.686 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2c0806ae70 msgr2=0x7f2c0806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2c0806ae70 0x7f2c0806d330 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c036000 tx=0x7f2c0c03c040 comp rx=0 tx=0).stop 2023-12-16T21:15:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 msgr2=0x7f2c1c148750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 0x7f2c1c148750 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c400 tx=0x7f2c1000c8c0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 shutdown_connections 2023-12-16T21:15:06.688 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2c0806ae70 0x7f2c0806d330 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c1c0d3900 0x7f2c1c148750 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1c15d330 0x7f2c1c0c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 --2- 172.21.15.46:0/2794998968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1c0d2910 0x7f2c1c0cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.687+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 >> 172.21.15.46:0/2794998968 conn(0x7f2c1c072020 msgr2=0x7f2c1c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.688+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 shutdown_connections 2023-12-16T21:15:06.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.688+0000 7f2c22089640 1 -- 172.21.15.46:0/2794998968 wait complete. 2023-12-16T21:15:06.699 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:06.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.808+0000 7f8b438da640 1 Processor -- start 2023-12-16T21:15:06.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.808+0000 7f8b438da640 1 -- start start 2023-12-16T21:15:06.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b438da640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8b3c06ea50 0x7f8b3c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b438da640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c161d20 con 0x7f8b3c0d0ed0 2023-12-16T21:15:06.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b438da640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b3c06f370 con 0x7f8b3c160fb0 2023-12-16T21:15:06.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b438da640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3c15e760 con 0x7f8b3c06ea50 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b41e77640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8b3c0d0ed0 0x7f8b3c0d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58472/0 (socket says 172.21.15.46:58472) 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b41e77640 1 -- 172.21.15.46:0/3464429482 learned_addr learned my addr 172.21.15.46:0/3464429482 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b41676640 1 --2- 172.21.15.46:0/3464429482 >> v2:172.21.15.121:3300/0 conn(0x7f8b3c06ea50 0x7f8b3c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092367921 0 0) 0x7f8b3c161d20 con 0x7f8b3c0d0ed0 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b24003520 con 0x7f8b3c0d0ed0 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3015883529 0 0) 0x7f8b24003520 con 0x7f8b3c0d0ed0 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 >> v2:172.21.15.121:3300/0 conn(0x7f8b3c06ea50 msgr2=0x7f8b3c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 --2- 172.21.15.46:0/3464429482 >> v2:172.21.15.121:3300/0 conn(0x7f8b3c06ea50 0x7f8b3c06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 >> v1:172.21.15.121:6789/0 conn(0x7f8b3c160fb0 legacy=0x7f8b3c161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c15d9a0 con 0x7f8b3c0d0ed0 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8b2c0019f0 con 0x7f8b3c0d0ed0 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b41676640 1 --2- 172.21.15.46:0/3464429482 >> v2:172.21.15.121:3300/0 conn(0x7f8b3c06ea50 0x7f8b3c06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:06.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 >> v1:172.21.15.46:6789/0 conn(0x7f8b3c0d0ed0 legacy=0x7f8b3c0d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b24003970 0x7f8b24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b240047c0 0x7f8b24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b2400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15e760 con 0x7f8b24005180 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2400c120 con 0x7f8b240047c0 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.809+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2400c2e0 con 0x7f8b24003970 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41e77640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b2400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b40e75640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b240047c0 0x7f8b24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41676640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b24003970 0x7f8b24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41e77640 1 -- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b24003970 msgr2=0x7f8b24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41e77640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b24003970 0x7f8b24003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.814 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41e77640 1 -- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b240047c0 msgr2=0x7f8b24004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41e77640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b240047c0 0x7f8b24004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.810+0000 7f8b41e77640 1 -- 172.21.15.46:0/3464429482 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b24003520 con 0x7f8b24005180 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b41e77640 1 -- 172.21.15.46:0/3464429482 reap_dead start 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b41e77640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b2400b980 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00a830 tx=0x7f8b240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf35c52dc98f061 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0072d0 con 0x7f8b24005180 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c02f070 con 0x7f8b24005180 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b3bfff640 1 -- 172.21.15.46:0/3464429482 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0076b0 con 0x7f8b24005180 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 -- 172.21.15.46:0/3464429482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 msgr2=0x7f8b2400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b2400b980 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00a830 tx=0x7f8b240057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 -- 172.21.15.46:0/3464429482 shutdown_connections 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 --2- 172.21.15.46:0/3464429482 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b2400b980 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 -- 172.21.15.46:0/3464429482 >> 172.21.15.46:0/3464429482 conn(0x7f8b3c06a6b0 msgr2=0x7f8b3c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 -- 172.21.15.46:0/3464429482 shutdown_connections 2023-12-16T21:15:06.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.811+0000 7f8b438da640 1 -- 172.21.15.46:0/3464429482 wait complete. 2023-12-16T21:15:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 Processor -- start 2023-12-16T21:15:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 -- start start 2023-12-16T21:15:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b3c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b3c15a030 0x7f8b3c15a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b3c139ba0 0x7f8b3c06cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c15ada0 con 0x7f8b24005180 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c159ba0 con 0x7f8b3c15a030 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b438da640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3c158f90 con 0x7f8b3c139ba0 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b41676640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b3c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b41676640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b3c1588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35142/0 (socket says 172.21.15.46:35142) 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.813+0000 7f8b41676640 1 -- 172.21.15.46:0/1887416754 learned_addr learned my addr 172.21.15.46:0/1887416754 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41e77640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b3c139ba0 0x7f8b3c06cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b40e75640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b3c15a030 0x7f8b3c15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41676640 1 -- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b3c139ba0 msgr2=0x7f8b3c06cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41676640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b3c139ba0 0x7f8b3c06cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.819 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41676640 1 -- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b3c15a030 msgr2=0x7f8b3c15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41676640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b3c15a030 0x7f8b3c15a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41676640 1 -- 172.21.15.46:0/1887416754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c159270 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b41676640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b3c1588d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b34011a90 tx=0x7f8b34011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3400fbb0 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3400fd10 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34004260 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3c159550 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.814+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3c13a020 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.815+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3c069400 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.816+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8b34003610 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.816+0000 7f8b39ffb640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b2806ae70 0x7f8b2806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.816+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8b34089de0 con 0x7f8b24005180 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.816+0000 7f8b40e75640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b2806ae70 0x7f8b2806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:06.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.817+0000 7f8b40e75640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b2806ae70 0x7f8b2806d330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:06.821 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.820+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8b34054d00 con 0x7f8b24005180 2023-12-16T21:15:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.994+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8b3c0676d0 con 0x7f8b24005180 2023-12-16T21:15:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.995+0000 7f8b39ffb640 1 -- 172.21.15.46:0/1887416754 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8b34022020 con 0x7f8b24005180 2023-12-16T21:15:06.995 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.996+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b2806ae70 msgr2=0x7f8b2806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.996+0000 7f8b438da640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b2806ae70 0x7f8b2806d330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b30000c00 tx=0x7f8b30009040 comp rx=0 tx=0).stop 2023-12-16T21:15:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.996+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 msgr2=0x7f8b3c1588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.996+0000 7f8b438da640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b3c1588d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8b34011a90 tx=0x7f8b34011f50 comp rx=0 tx=0).stop 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 shutdown_connections 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8b2806ae70 0x7f8b2806d330 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b3c139ba0 0x7f8b3c06cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b3c15a030 0x7f8b3c15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 --2- 172.21.15.46:0/1887416754 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8b24005180 0x7f8b3c1588d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 >> 172.21.15.46:0/1887416754 conn(0x7f8b3c06a6b0 msgr2=0x7f8b3c0b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 shutdown_connections 2023-12-16T21:15:06.998 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:06.997+0000 7f8b438da640 1 -- 172.21.15.46:0/1887416754 wait complete. 2023-12-16T21:15:07.010 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:15:07.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.118+0000 7f614b11f640 1 Processor -- start 2023-12-16T21:15:07.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f614b11f640 1 -- start start 2023-12-16T21:15:07.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f614b11f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f614415d330 0x7f61440b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f614b11f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f614b11f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61440b2b10 con 0x7f61440d2910 2023-12-16T21:15:07.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f614b11f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61440d1590 con 0x7f614415d330 2023-12-16T21:15:07.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f614b11f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61440d1870 con 0x7f61440d3900 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.119+0000 7f6148ebb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f614415d330 0x7f61440b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6148ebb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f614415d330 0x7f61440b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35146/0 (socket says 172.21.15.46:35146) 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6148ebb640 1 -- 172.21.15.46:0/918517248 learned_addr learned my addr 172.21.15.46:0/918517248 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6143fff640 1 --1- 172.21.15.46:0/918517248 >> v1:172.21.15.46:6789/0 conn(0x7f61440d2910 0x7f61440d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:58476/0 (socket says 172.21.15.46:58476) 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f61496bc640 1 --2- 172.21.15.46:0/918517248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6148ebb640 1 -- 172.21.15.46:0/918517248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 msgr2=0x7f61440b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6148ebb640 1 --2- 172.21.15.46:0/918517248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6148ebb640 1 -- 172.21.15.46:0/918517248 >> v1:172.21.15.46:6789/0 conn(0x7f61440d2910 legacy=0x7f61440d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f61437fe640 1 -- 172.21.15.46:0/918517248 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063088472 0 0) 0x7f61440b2b10 con 0x7f61440d2910 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.120+0000 7f6148ebb640 1 -- 172.21.15.46:0/918517248 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61440d1b50 con 0x7f614415d330 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f6148ebb640 1 --2- 172.21.15.46:0/918517248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f614415d330 0x7f61440b8b80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6138009870 tx=0x7f613802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a90f97bbdb09d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f61437fe640 1 -- 172.21.15.46:0/918517248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138031640 con 0x7f614415d330 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f61437fe640 1 -- 172.21.15.46:0/918517248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61380317a0 con 0x7f614415d330 2023-12-16T21:15:07.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f61437fe640 1 -- 172.21.15.46:0/918517248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138031b30 con 0x7f614415d330 2023-12-16T21:15:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 -- 172.21.15.46:0/918517248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f614415d330 msgr2=0x7f61440b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 --2- 172.21.15.46:0/918517248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f614415d330 0x7f61440b8b80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6138009870 tx=0x7f613802b8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 -- 172.21.15.46:0/918517248 shutdown_connections 2023-12-16T21:15:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 --2- 172.21.15.46:0/918517248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 --2- 172.21.15.46:0/918517248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f614415d330 0x7f61440b8b80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 -- 172.21.15.46:0/918517248 >> 172.21.15.46:0/918517248 conn(0x7f6144072020 msgr2=0x7f6144072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 -- 172.21.15.46:0/918517248 shutdown_connections 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.121+0000 7f614b11f640 1 -- 172.21.15.46:0/918517248 wait complete. 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.122+0000 7f614b11f640 1 Processor -- start 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 -- start start 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 0x7f614414b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614415d330 0x7f61440b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614415f260 con 0x7f61440d2910 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614415e650 con 0x7f614415d330 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.123+0000 7f614b11f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f614415e930 con 0x7f61440d3900 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 0x7f614414b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 0x7f614414b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35156/0 (socket says 172.21.15.46:35156) 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 -- 172.21.15.46:0/3828223344 learned_addr learned my addr 172.21.15.46:0/3828223344 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:07.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6143fff640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f61496bc640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614415d330 0x7f61440b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 -- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 msgr2=0x7f61440b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 -- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614415d330 msgr2=0x7f61440b8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614415d330 0x7f61440b8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 -- 172.21.15.46:0/3828223344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380094d0 con 0x7f61440d2910 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f6148ebb640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 0x7f614414b780 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6138009870 tx=0x7f61380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138004230 con 0x7f61440d2910 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138004390 con 0x7f61440d2910 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613803baa0 con 0x7f61440d2910 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614415ec10 con 0x7f61440d2910 2023-12-16T21:15:07.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.124+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614415d920 con 0x7f61440d2910 2023-12-16T21:15:07.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.125+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61440d2dc0 con 0x7f61440d2910 2023-12-16T21:15:07.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.126+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6138043070 con 0x7f61440d2910 2023-12-16T21:15:07.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.126+0000 7f61417fa640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f612806ae70 0x7f612806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.126+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f61380ab670 con 0x7f61440d2910 2023-12-16T21:15:07.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.127+0000 7f6143fff640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f612806ae70 0x7f612806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.127+0000 7f6143fff640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f612806ae70 0x7f612806d330 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f61441581e0 tx=0x7f6130009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:07.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.131+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6138076620 con 0x7f61440d2910 2023-12-16T21:15:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.306+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f6144069400 con 0x7f61440d2910 2023-12-16T21:15:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.306+0000 7f61417fa640 1 -- 172.21.15.46:0/3828223344 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7f6138079c40 con 0x7f61440d2910 2023-12-16T21:15:07.307 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:07.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.308+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f612806ae70 msgr2=0x7f612806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.308+0000 7f614b11f640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f612806ae70 0x7f612806d330 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f61441581e0 tx=0x7f6130009040 comp rx=0 tx=0).stop 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.308+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 msgr2=0x7f614414b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.308+0000 7f614b11f640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 0x7f614414b780 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6138009870 tx=0x7f61380046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 shutdown_connections 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f612806ae70 0x7f612806d330 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f614415d330 0x7f61440b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f61440d3900 0x7f61440b86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 --2- 172.21.15.46:0/3828223344 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61440d2910 0x7f614414b780 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 >> 172.21.15.46:0/3828223344 conn(0x7f6144072020 msgr2=0x7f6144072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 shutdown_connections 2023-12-16T21:15:07.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.309+0000 7f614b11f640 1 -- 172.21.15.46:0/3828223344 wait complete. 2023-12-16T21:15:07.320 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.427+0000 7ff86486b640 1 Processor -- start 2023-12-16T21:15:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.427+0000 7ff86486b640 1 -- start start 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff86486b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff860144980 0x7ff86015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff86486b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8600ceb70 0x7ff8600cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff86486b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff86486b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff860071770 con 0x7ff8600ceb70 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff86486b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8600b4ce0 con 0x7ff860144980 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff86486b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600ccdc0 con 0x7ff8600b4370 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8600ceb70 0x7ff8600cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8600ceb70 0x7ff8600cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35168/0 (socket says 172.21.15.46:35168) 2023-12-16T21:15:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 -- 172.21.15.46:0/3811740048 learned_addr learned my addr 172.21.15.46:0/3811740048 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85ed9d640 1 --2- 172.21.15.46:0/3811740048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85e59c640 1 --2- 172.21.15.46:0/3811740048 >> v2:172.21.15.121:3300/0 conn(0x7ff860144980 0x7ff86015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 -- 172.21.15.46:0/3811740048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 msgr2=0x7ff8600b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 --2- 172.21.15.46:0/3811740048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 -- 172.21.15.46:0/3811740048 >> v2:172.21.15.121:3300/0 conn(0x7ff860144980 msgr2=0x7ff86015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 --2- 172.21.15.46:0/3811740048 >> v2:172.21.15.121:3300/0 conn(0x7ff860144980 0x7ff86015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.428+0000 7ff85dd9b640 1 -- 172.21.15.46:0/3811740048 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8600cd0a0 con 0x7ff8600ceb70 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.429+0000 7ff85dd9b640 1 --2- 172.21.15.46:0/3811740048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff8600cc450 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff8600b4ec0 tx=0x7ff84802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e3f60ec2d24dbd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.429+0000 7ff85d59a640 1 -- 172.21.15.46:0/3811740048 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848031870 con 0x7ff8600ceb70 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.429+0000 7ff85d59a640 1 -- 172.21.15.46:0/3811740048 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8480319d0 con 0x7ff8600ceb70 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.429+0000 7ff85d59a640 1 -- 172.21.15.46:0/3811740048 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848031d60 con 0x7ff8600ceb70 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 -- 172.21.15.46:0/3811740048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 msgr2=0x7ff8600cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 --2- 172.21.15.46:0/3811740048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff8600cc450 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff8600b4ec0 tx=0x7ff84802b910 comp rx=0 tx=0).stop 2023-12-16T21:15:07.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 -- 172.21.15.46:0/3811740048 shutdown_connections 2023-12-16T21:15:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 --2- 172.21.15.46:0/3811740048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 --2- 172.21.15.46:0/3811740048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff8600cc450 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 --2- 172.21.15.46:0/3811740048 >> v2:172.21.15.121:3300/0 conn(0x7ff860144980 0x7ff86015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 -- 172.21.15.46:0/3811740048 >> 172.21.15.46:0/3811740048 conn(0x7ff86006a4d0 msgr2=0x7ff86006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 -- 172.21.15.46:0/3811740048 shutdown_connections 2023-12-16T21:15:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.430+0000 7ff86486b640 1 -- 172.21.15.46:0/3811740048 wait complete. 2023-12-16T21:15:07.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.431+0000 7ff86486b640 1 Processor -- start 2023-12-16T21:15:07.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.431+0000 7ff86486b640 1 -- start start 2023-12-16T21:15:07.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff86486b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff86486b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff86013f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff86486b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff860144980 0x7ff8601309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff86486b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600afc90 con 0x7ff8600ceb70 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff86486b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600aec90 con 0x7ff860144980 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff86486b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff86013dcc0 con 0x7ff8600b4370 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff86013f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff86013f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35182/0 (socket says 172.21.15.46:35182) 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 -- 172.21.15.46:0/2675441845 learned_addr learned my addr 172.21.15.46:0/2675441845 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85e59c640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85ed9d640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff860144980 0x7ff8601309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 -- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 msgr2=0x7ff8600af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 -- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff860144980 msgr2=0x7ff8601309f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff860144980 0x7ff8601309f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.432+0000 7ff85dd9b640 1 -- 172.21.15.46:0/2675441845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8480094d0 con 0x7ff8600ceb70 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.433+0000 7ff85dd9b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff86013f960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff84802b950 tx=0x7ff8480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.433+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84803ad30 con 0x7ff8600ceb70 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.433+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8480443f0 con 0x7ff8600ceb70 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.433+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff860131540 con 0x7ff8600ceb70 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.433+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8480447a0 con 0x7ff8600ceb70 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.433+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8600aef50 con 0x7ff8600ceb70 2023-12-16T21:15:07.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.434+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff860069400 con 0x7ff8600ceb70 2023-12-16T21:15:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.434+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff84804d450 con 0x7ff8600ceb70 2023-12-16T21:15:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.435+0000 7ff856ffd640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff84006ae90 0x7ff84006d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.435+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff84802d040 con 0x7ff8600ceb70 2023-12-16T21:15:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.435+0000 7ff85e59c640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff84006ae90 0x7ff84006d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:07.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.436+0000 7ff85e59c640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff84006ae90 0x7ff84006d350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff858006fd0 tx=0x7ff858008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:07.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.439+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff848076ce0 con 0x7ff8600ceb70 2023-12-16T21:15:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.631+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff86015ca30 con 0x7ff8600ceb70 2023-12-16T21:15:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.632+0000 7ff856ffd640 1 -- 172.21.15.46:0/2675441845 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v37) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff848035020 con 0x7ff8600ceb70 2023-12-16T21:15:07.632 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.633+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff84006ae90 msgr2=0x7ff84006d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.633+0000 7ff86486b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff84006ae90 0x7ff84006d350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff858006fd0 tx=0x7ff858008040 comp rx=0 tx=0).stop 2023-12-16T21:15:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 msgr2=0x7ff86013f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:07.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff86013f960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff84802b950 tx=0x7ff8480046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 shutdown_connections 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff84006ae90 0x7ff84006d350 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8600ceb70 0x7ff86013f960 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff860144980 0x7ff8601309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 --2- 172.21.15.46:0/2675441845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8600b4370 0x7ff8600af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 >> 172.21.15.46:0/2675441845 conn(0x7ff86006a4d0 msgr2=0x7ff86015b900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 shutdown_connections 2023-12-16T21:15:07.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:07.634+0000 7ff86486b640 1 -- 172.21.15.46:0/2675441845 wait complete. 2023-12-16T21:15:07.645 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:15:07.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:07.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:07.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[106451]: fuse finished with error 0 and tester_r 0 2023-12-16T21:15:07.676 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:13.777 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:13.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:13.778 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:15:13.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2023-12-16T21:15:13.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.962+0000 7fa1083d5640 1 Processor -- start 2023-12-16T21:15:13.963 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.962+0000 7fa1083d5640 1 -- start start 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.963+0000 7fa1083d5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10006bbf0 0x7fa1000b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.963+0000 7fa1083d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa1000ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa1083d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa100146d50 con 0x7fa100140480 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa1083d5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000ada30 con 0x7fa10006bbf0 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa1083d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000b37c0 con 0x7fa1001464d0 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10006bbf0 0x7fa1000b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa10006bbf0 0x7fa1000b2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52002/0 (socket says 172.21.15.46:52002) 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 -- 172.21.15.46:0/1935413425 learned_addr learned my addr 172.21.15.46:0/1935413425 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa106972640 1 --2- 172.21.15.46:0/1935413425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa1000ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 -- 172.21.15.46:0/1935413425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 msgr2=0x7fa1000ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 --2- 172.21.15.46:0/1935413425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa1000ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 -- 172.21.15.46:0/1935413425 >> v1:172.21.15.121:6789/0 conn(0x7fa100140480 legacy=0x7fa10006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.964+0000 7fa105970640 1 -- 172.21.15.46:0/1935413425 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000ad700 con 0x7fa10006bbf0 2023-12-16T21:15:13.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.965+0000 7fa105970640 1 --2- 172.21.15.46:0/1935413425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006bbf0 0x7fa1000b2820 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009870 tx=0x7fa0f402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a95e6f10483ca3d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:13.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.965+0000 7fa10516f640 1 -- 172.21.15.46:0/1935413425 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40317c0 con 0x7fa10006bbf0 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.965+0000 7fa10516f640 1 -- 172.21.15.46:0/1935413425 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f4031920 con 0x7fa10006bbf0 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.965+0000 7fa10516f640 1 -- 172.21.15.46:0/1935413425 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4031cb0 con 0x7fa10006bbf0 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.965+0000 7fa1083d5640 1 -- 172.21.15.46:0/1935413425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006bbf0 msgr2=0x7fa1000b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.965+0000 7fa1083d5640 1 --2- 172.21.15.46:0/1935413425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006bbf0 0x7fa1000b2820 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4009870 tx=0x7fa0f402b860 comp rx=0 tx=0).stop 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.966+0000 7fa1083d5640 1 -- 172.21.15.46:0/1935413425 shutdown_connections 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.966+0000 7fa1083d5640 1 --2- 172.21.15.46:0/1935413425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa1000ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.966+0000 7fa1083d5640 1 --2- 172.21.15.46:0/1935413425 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa10006bbf0 0x7fa1000b2820 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.966+0000 7fa1083d5640 1 -- 172.21.15.46:0/1935413425 >> 172.21.15.46:0/1935413425 conn(0x7fa100070c70 msgr2=0x7fa100071050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.966+0000 7fa1083d5640 1 -- 172.21.15.46:0/1935413425 shutdown_connections 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.966+0000 7fa1083d5640 1 -- 172.21.15.46:0/1935413425 wait complete. 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.967+0000 7fa1083d5640 1 Processor -- start 2023-12-16T21:15:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.967+0000 7fa1083d5640 1 -- start start 2023-12-16T21:15:13.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10006bbf0 0x7fa10015c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 0x7fa100135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa100136d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100135c80 con 0x7fa100140480 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100135f60 con 0x7fa10006bbf0 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa100136240 con 0x7fa1001464d0 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 0x7fa100135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 0x7fa100135380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52008/0 (socket says 172.21.15.46:52008) 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 -- 172.21.15.46:0/2670692153 learned_addr learned my addr 172.21.15.46:0/2670692153 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa106171640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10006bbf0 0x7fa10015c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 -- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 msgr2=0x7fa100136d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa100136d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 -- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10006bbf0 msgr2=0x7fa10015c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10006bbf0 0x7fa10015c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 -- 172.21.15.46:0/2670692153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f40094d0 con 0x7fa100140480 2023-12-16T21:15:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa105970640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 0x7fa100135380 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4000c00 tx=0x7fa0f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40329a0 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f4032b20 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa100136520 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.968+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40453f0 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.969+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001358c0 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.969+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa100139950 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.970+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa0f404d3f0 con 0x7fa100140480 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.970+0000 7fa0feffd640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0ec06ae40 0x7fa0ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.970+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fa0f40ab770 con 0x7fa100140480 2023-12-16T21:15:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.971+0000 7fa106171640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0ec06ae40 0x7fa0ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.971+0000 7fa106171640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0ec06ae40 0x7fa0ec06d300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa10015d8f0 tx=0x7fa0f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:13.975 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:13.975+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa0f4076690 con 0x7fa100140480 2023-12-16T21:15:14.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.218+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fa100069400 con 0x7fa100140480 2023-12-16T21:15:14.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.219+0000 7fa0feffd640 1 -- 172.21.15.46:0/2670692153 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v18) v1 ==== 184+0+141 (secure 0 0 0) 0x7fa0f4079cb0 con 0x7fa100140480 2023-12-16T21:15:14.220 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:15:14.220 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:15:14.220 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2023-12-16T21:15:14.220 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:15:14.220 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw pool=cephfs_data" 2023-12-16T21:15:14.221 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:15:14.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0ec06ae40 msgr2=0x7fa0ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0ec06ae40 0x7fa0ec06d300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa10015d8f0 tx=0x7fa0f0009040 comp rx=0 tx=0).stop 2023-12-16T21:15:14.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 msgr2=0x7fa100135380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 0x7fa100135380 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4000c00 tx=0x7fa0f40046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 shutdown_connections 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa0ec06ae40 0x7fa0ec06d300 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa100140480 0x7fa100135380 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1001464d0 0x7fa100136d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 --2- 172.21.15.46:0/2670692153 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa10006bbf0 0x7fa10015c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 >> 172.21.15.46:0/2670692153 conn(0x7fa100070c70 msgr2=0x7fa1000ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 shutdown_connections 2023-12-16T21:15:14.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.222+0000 7fa1083d5640 1 -- 172.21.15.46:0/2670692153 wait complete. 2023-12-16T21:15:14.233 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-16T21:15:14.233 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:15:14.277 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:15:14.335 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-16T21:15:14.335 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:14.335 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:14.350 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:15:14.350 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:14.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:14.351 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:15:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:14.406 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-16T21:15:14.406 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:15:14.406 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:14.424 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:15:14.424 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:15:14.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:14.425 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-16T21:15:14.481 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-16T21:15:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:14.483 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:14.483 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:14.498 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:15:14.498 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:14.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:14.498 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-16T21:15:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-16T21:15:14.554 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:15:14.554 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:14.572 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:15:14.572 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:15:14.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:14.572 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-16T21:15:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:14.630 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-16T21:15:14.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.736+0000 7fc6a1675640 1 Processor -- start 2023-12-16T21:15:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.736+0000 7fc6a1675640 1 -- start start 2023-12-16T21:15:14.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc6a1675640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 0x7fc69c166000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc6a1675640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c15ba40 con 0x7fc69c15c2f0 2023-12-16T21:15:14.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc6a1675640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69c1416e0 con 0x7fc69c06f210 2023-12-16T21:15:14.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc6a1675640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c15c6d0 con 0x7fc69c140340 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc69affd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc69c15c2f0 0x7fc69c06eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36842/0 (socket says 172.21.15.46:36842) 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc69affd640 1 -- 172.21.15.46:0/3688164573 learned_addr learned my addr 172.21.15.46:0/3688164573 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc69b7fe640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 0x7fc69c166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589337379 0 0) 0x7fc69c15ba40 con 0x7fc69c15c2f0 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.737+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc688003520 con 0x7fc69c15c2f0 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528720868 0 0) 0x7fc69c1416e0 con 0x7fc69c06f210 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c15ba40 con 0x7fc69c06f210 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280402623 0 0) 0x7fc688003520 con 0x7fc69c15c2f0 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 msgr2=0x7fc69c166000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 0x7fc69c166000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 >> v1:172.21.15.121:6789/0 conn(0x7fc69c06f210 legacy=0x7fc69c140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c141400 con 0x7fc69c15c2f0 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fc694001670 con 0x7fc69c15c2f0 2023-12-16T21:15:14.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 >> v1:172.21.15.46:6789/0 conn(0x7fc69c15c2f0 legacy=0x7fc69c06eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc688003b30 0x7fc688003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 0x7fc6880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc688004d00 0x7fc688005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68800ba30 con 0x7fc688004d00 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68800bb70 con 0x7fc688003b30 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc68800bce0 con 0x7fc69c140340 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69b7fe640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc688004d00 0x7fc688005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69affd640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc688003b30 0x7fc688003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69a7fc640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 0x7fc6880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69b7fe640 1 -- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 msgr2=0x7fc6880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69b7fe640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c140340 0x7fc6880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69b7fe640 1 -- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc688003b30 msgr2=0x7fc688003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69b7fe640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc688003b30 0x7fc688003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.738+0000 7fc69b7fe640 1 -- 172.21.15.46:0/3688164573 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc688003520 con 0x7fc688004d00 2023-12-16T21:15:14.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.739+0000 7fc69b7fe640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc688004d00 0x7fc688005180 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009f70 tx=0x7fc6880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a047b95608954675 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.739+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c02cc90 con 0x7fc688004d00 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.739+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c00f040 con 0x7fc688004d00 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.739+0000 7fc699ffb640 1 -- 172.21.15.46:0/3688164573 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c034440 con 0x7fc688004d00 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc6a1675640 1 -- 172.21.15.46:0/3688164573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc688004d00 msgr2=0x7fc688005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3688164573 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc688004d00 0x7fc688005180 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009f70 tx=0x7fc6880057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc69b7fe640 1 -- 172.21.15.46:0/3688164573 reap_dead start 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc6a1675640 1 -- 172.21.15.46:0/3688164573 shutdown_connections 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc6a1675640 1 -- 172.21.15.46:0/3688164573 >> 172.21.15.46:0/3688164573 conn(0x7fc69c06a6b0 msgr2=0x7fc69c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc6a1675640 1 -- 172.21.15.46:0/3688164573 shutdown_connections 2023-12-16T21:15:14.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.740+0000 7fc6a1675640 1 -- 172.21.15.46:0/3688164573 wait complete. 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.741+0000 7fc6a1675640 1 Processor -- start 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.741+0000 7fc6a1675640 1 -- start start 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc6a1675640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 0x7fc69c0ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc6a1675640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c0acef0 0x7fc69c1f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc6a1675640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc69c15f0c0 0x7fc69c1f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc6a1675640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c0b3a50 con 0x7fc69c15ece0 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc6a1675640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c1f5160 con 0x7fc69c15f0c0 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc6a1675640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c1f4e80 con 0x7fc69c0acef0 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 0x7fc69c0ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 0x7fc69c0ac9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52028/0 (socket says 172.21.15.46:52028) 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 -- 172.21.15.46:0/3323389376 learned_addr learned my addr 172.21.15.46:0/3323389376 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:14.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69a7fc640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c0acef0 0x7fc69c1f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69b7fe640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc69c15f0c0 0x7fc69c1f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 -- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c0acef0 msgr2=0x7fc69c1f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c0acef0 0x7fc69c1f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 -- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc69c15f0c0 msgr2=0x7fc69c1f47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc69c15f0c0 0x7fc69c1f47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.742+0000 7fc69affd640 1 -- 172.21.15.46:0/3323389376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c0a8830 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.743+0000 7fc69affd640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 0x7fc69c0ac9b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc694007fd0 tx=0x7fc6940074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.743+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694001b10 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.743+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc694007740 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.743+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69c0a8b10 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.743+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69400fc00 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.743+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69c0a9310 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.744+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69c0a9030 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.744+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc69400fd60 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.745+0000 7fc6937fe640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc67c06ae40 0x7fc67c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.745+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fc694089950 con 0x7fc69c15ece0 2023-12-16T21:15:14.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.745+0000 7fc69a7fc640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc67c06ae40 0x7fc67c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:14.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.745+0000 7fc69a7fc640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc67c06ae40 0x7fc67c06d300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0ad2f0 tx=0x7fc684009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:14.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.749+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc69c0a9030 con 0x7fc69c15ece0 2023-12-16T21:15:14.960 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.960+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc69c0a9030 con 0x7fc69c15ece0 2023-12-16T21:15:14.961 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.960+0000 7fc6937fe640 1 -- 172.21.15.46:0/3323389376 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fc69c0a9030 con 0x7fc69c15ece0 2023-12-16T21:15:14.961 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:14.961 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:44.132960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:15:14.962 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc67c06ae40 msgr2=0x7fc67c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc67c06ae40 0x7fc67c06d300 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0ad2f0 tx=0x7fc684009040 comp rx=0 tx=0).stop 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 msgr2=0x7fc69c0ac9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 0x7fc69c0ac9b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fc694007fd0 tx=0x7fc6940074f0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 shutdown_connections 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc67c06ae40 0x7fc67c06d300 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc69c15f0c0 0x7fc69c1f47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc69c0acef0 0x7fc69c1f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 --2- 172.21.15.46:0/3323389376 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc69c15ece0 0x7fc69c0ac9b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.962+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 >> 172.21.15.46:0/3323389376 conn(0x7fc69c06a6b0 msgr2=0x7fc69c0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.963+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 shutdown_connections 2023-12-16T21:15:14.964 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:14.963+0000 7fc6a1675640 1 -- 172.21.15.46:0/3323389376 wait complete. 2023-12-16T21:15:14.975 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.082+0000 7ff09b089640 1 Processor -- start 2023-12-16T21:15:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.082+0000 7ff09b089640 1 -- start start 2023-12-16T21:15:15.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.082+0000 7ff09b089640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0940d3900 0x7ff0940b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.082+0000 7ff09b089640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940b2b10 con 0x7ff0940d2910 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.082+0000 7ff09b089640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940d1590 con 0x7ff09415d330 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.082+0000 7ff09b089640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0940d1870 con 0x7ff0940d3900 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff093fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0940d2910 0x7ff0940d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36856/0 (socket says 172.21.15.46:36856) 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff093fff640 1 -- 172.21.15.46:0/2230468298 learned_addr learned my addr 172.21.15.46:0/2230468298 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff099626640 1 --2- 172.21.15.46:0/2230468298 >> v2:172.21.15.121:3300/0 conn(0x7ff0940d3900 0x7ff0940b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff0937fe640 1 -- 172.21.15.46:0/2230468298 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581547550 0 0) 0x7ff0940b2b10 con 0x7ff0940d2910 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff0937fe640 1 -- 172.21.15.46:0/2230468298 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff084003520 con 0x7ff0940d2910 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff099626640 1 -- 172.21.15.46:0/2230468298 >> v1:172.21.15.121:6789/0 conn(0x7ff09415d330 legacy=0x7ff0940b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff099626640 1 -- 172.21.15.46:0/2230468298 >> v1:172.21.15.46:6789/0 conn(0x7ff0940d2910 legacy=0x7ff0940d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff0937fe640 1 -- 172.21.15.46:0/2230468298 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553949635 0 0) 0x7ff084003520 con 0x7ff0940d2910 2023-12-16T21:15:15.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.083+0000 7ff099626640 1 -- 172.21.15.46:0/2230468298 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0940d1b50 con 0x7ff0940d3900 2023-12-16T21:15:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.084+0000 7ff099626640 1 --2- 172.21.15.46:0/2230468298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0940d3900 0x7ff0940b6390 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009d90 tx=0x7ff08c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c15f469fa6b9529 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.084+0000 7ff0937fe640 1 -- 172.21.15.46:0/2230468298 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080001670 con 0x7ff0940d3900 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.084+0000 7ff0937fe640 1 -- 172.21.15.46:0/2230468298 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c02c8b0 con 0x7ff0940d3900 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.084+0000 7ff0937fe640 1 -- 172.21.15.46:0/2230468298 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c02cc20 con 0x7ff0940d3900 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 -- 172.21.15.46:0/2230468298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0940d3900 msgr2=0x7ff0940b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 --2- 172.21.15.46:0/2230468298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0940d3900 0x7ff0940b6390 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009d90 tx=0x7ff08c02bd80 comp rx=0 tx=0).stop 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 -- 172.21.15.46:0/2230468298 shutdown_connections 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 --2- 172.21.15.46:0/2230468298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0940d3900 0x7ff0940b6390 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 -- 172.21.15.46:0/2230468298 >> 172.21.15.46:0/2230468298 conn(0x7ff094072020 msgr2=0x7ff094072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 -- 172.21.15.46:0/2230468298 shutdown_connections 2023-12-16T21:15:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.085+0000 7ff09b089640 1 -- 172.21.15.46:0/2230468298 wait complete. 2023-12-16T21:15:15.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.086+0000 7ff09b089640 1 Processor -- start 2023-12-16T21:15:15.087 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.086+0000 7ff09b089640 1 -- start start 2023-12-16T21:15:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff09b089640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0940d2910 0x7ff0941f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff09b089640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 0x7ff0940c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff09b089640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09415d330 0x7ff0940b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff09b089640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940c6d40 con 0x7ff0940d3900 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff09b089640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940c7020 con 0x7ff09415d330 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff09b089640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0940c7300 con 0x7ff0940d2910 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 0x7ff0940c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 0x7ff0940c91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52038/0 (socket says 172.21.15.46:52038) 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 -- 172.21.15.46:0/859088315 learned_addr learned my addr 172.21.15.46:0/859088315 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff098e25640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0940d2910 0x7ff0941f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff099626640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09415d330 0x7ff0940b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 -- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0940d2910 msgr2=0x7ff0941f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0940d2910 0x7ff0941f63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 -- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09415d330 msgr2=0x7ff0940b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09415d330 0x7ff0940b0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 -- 172.21.15.46:0/859088315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c009a70 con 0x7ff0940d3900 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.087+0000 7ff093fff640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 0x7ff0940c91c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff0800011e0 tx=0x7ff08000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.088+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080005ad0 con 0x7ff0940d3900 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.088+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080005c50 con 0x7ff0940d3900 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.088+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0940c75e0 con 0x7ff0940d3900 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.088+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0800072b0 con 0x7ff0940d3900 2023-12-16T21:15:15.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.088+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0940c7de0 con 0x7ff0940d3900 2023-12-16T21:15:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.089+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0940c7b00 con 0x7ff0940d3900 2023-12-16T21:15:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.089+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff080007410 con 0x7ff0940d3900 2023-12-16T21:15:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.090+0000 7ff0917fa640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07806ae40 0x7ff07806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.090+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff080088ef0 con 0x7ff0940d3900 2023-12-16T21:15:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.090+0000 7ff098e25640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07806ae40 0x7ff07806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.090+0000 7ff098e25640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07806ae40 0x7ff07806d300 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff088000f60 tx=0x7ff088009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.094+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff0940c7b00 con 0x7ff0940d3900 2023-12-16T21:15:15.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.269+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0940c7b00 con 0x7ff0940d3900 2023-12-16T21:15:15.271 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.270+0000 7ff0917fa640 1 -- 172.21.15.46:0/859088315 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7ff0940c7b00 con 0x7ff0940d3900 2023-12-16T21:15:15.272 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:15.272 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07806ae40 msgr2=0x7ff07806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07806ae40 0x7ff07806d300 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff088000f60 tx=0x7ff088009040 comp rx=0 tx=0).stop 2023-12-16T21:15:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 msgr2=0x7ff0940c91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 0x7ff0940c91c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff0800011e0 tx=0x7ff08000cbe0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 shutdown_connections 2023-12-16T21:15:15.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff07806ae40 0x7ff07806d300 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0940d3900 0x7ff0940c91c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff09415d330 0x7ff0940b0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 --2- 172.21.15.46:0/859088315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0940d2910 0x7ff0941f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 >> 172.21.15.46:0/859088315 conn(0x7ff094072020 msgr2=0x7ff094072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 shutdown_connections 2023-12-16T21:15:15.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.272+0000 7ff09b089640 1 -- 172.21.15.46:0/859088315 wait complete. 2023-12-16T21:15:15.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:14:34.660193+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:14:35.078326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-16T21:14:43.306876+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:15.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.396+0000 7f704ecb4640 1 Processor -- start 2023-12-16T21:15:15.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.396+0000 7f704ecb4640 1 -- start start 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ecb4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70480ac370 0x7f70480ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ecb4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7048071c80 0x7f7048072110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ecb4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70480adbb0 con 0x7f7048158ba0 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ecb4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7048072650 con 0x7f7048071c80 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ecb4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70480b4520 con 0x7f70480ac370 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ca50640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7048158ba0 0x7f704815c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36872/0 (socket says 172.21.15.46:36872) 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704ca50640 1 -- 172.21.15.46:0/4239642891 learned_addr learned my addr 172.21.15.46:0/4239642891 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f704d251640 1 --2- 172.21.15.46:0/4239642891 >> v2:172.21.15.121:3300/0 conn(0x7f7048071c80 0x7f7048072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.397+0000 7f7047fff640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70480ac370 0x7f70480ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212625794 0 0) 0x7f70480adbb0 con 0x7f7048158ba0 2023-12-16T21:15:15.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7038003520 con 0x7f7048158ba0 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1008153696 0 0) 0x7f7038003520 con 0x7f7048158ba0 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70480ac370 msgr2=0x7f70480ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70480ac370 0x7f70480ae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 >> v2:172.21.15.121:3300/0 conn(0x7f7048071c80 msgr2=0x7f7048072110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 --2- 172.21.15.46:0/4239642891 >> v2:172.21.15.121:3300/0 conn(0x7f7048071c80 0x7f7048072110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70480b3f60 con 0x7f7048158ba0 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f703c001670 con 0x7f7048158ba0 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 --2- 172.21.15.46:0/4239642891 >> v2:172.21.15.121:3300/0 conn(0x7f7048071c80 0x7f7048072110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 >> v1:172.21.15.46:6789/0 conn(0x7f7048158ba0 legacy=0x7f704815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7038003970 0x7f7038003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70380047c0 0x7f7038004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7038005180 0x7f703800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7048072650 con 0x7f7038005180 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480b4520 con 0x7f7038003970 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703800bf10 con 0x7f70380047c0 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f7047fff640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70480ac370 0x7f70480ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704ca50640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7038003970 0x7f7038003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7038005180 0x7f703800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f7047fff640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70380047c0 0x7f7038004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 -- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70380047c0 msgr2=0x7f7038004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70380047c0 0x7f7038004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 -- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7038003970 msgr2=0x7f7038003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7038003970 0x7f7038003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f704d251640 1 -- 172.21.15.46:0/4239642891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038003520 con 0x7f7038005180 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.398+0000 7f7047fff640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70380047c0 0x7f7038004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.399+0000 7f704d251640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7038005180 0x7f703800b980 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f704000e380 tx=0x7f70380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2bf3ef008b457e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.399+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704002cde0 con 0x7f7038005180 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.399+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704000f040 con 0x7f7038005180 2023-12-16T21:15:15.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.399+0000 7f70477fe640 1 -- 172.21.15.46:0/4239642891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040034640 con 0x7f7038005180 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704ecb4640 1 -- 172.21.15.46:0/4239642891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7038005180 msgr2=0x7f703800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704ecb4640 1 --2- 172.21.15.46:0/4239642891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7038005180 0x7f703800b980 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f704000e380 tx=0x7f70380057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704d251640 1 -- 172.21.15.46:0/4239642891 reap_dead start 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704ecb4640 1 -- 172.21.15.46:0/4239642891 shutdown_connections 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704ecb4640 1 -- 172.21.15.46:0/4239642891 >> 172.21.15.46:0/4239642891 conn(0x7f70480b4720 msgr2=0x7f70480b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704ecb4640 1 -- 172.21.15.46:0/4239642891 shutdown_connections 2023-12-16T21:15:15.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.400+0000 7f704ecb4640 1 -- 172.21.15.46:0/4239642891 wait complete. 2023-12-16T21:15:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.401+0000 7f704ecb4640 1 Processor -- start 2023-12-16T21:15:15.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.401+0000 7f704ecb4640 1 -- start start 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ecb4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7048098910 0x7f704806db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ecb4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7048098cf0 0x7f704815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ecb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 0x7f704815d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ecb4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704815de80 con 0x7f704815d490 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ecb4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704815c190 con 0x7f7048098cf0 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ecb4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704815c470 con 0x7f7048098910 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 0x7f704815d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 0x7f704815d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52052/0 (socket says 172.21.15.46:52052) 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 -- 172.21.15.46:0/1242691399 learned_addr learned my addr 172.21.15.46:0/1242691399 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f7047fff640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7048098cf0 0x7f704815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704ca50640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7048098910 0x7f704806db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 -- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7048098910 msgr2=0x7f704806db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7048098910 0x7f704806db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 -- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7048098cf0 msgr2=0x7f704815cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7048098cf0 0x7f704815cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.402+0000 7f704d251640 1 -- 172.21.15.46:0/1242691399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704815c750 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.403+0000 7f704d251640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 0x7f704815d940 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7040037040 tx=0x7f7040009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.403+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040043070 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.403+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7040034bd0 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.403+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704815ca30 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.403+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040003990 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.403+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70480b0520 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.404+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70480cfbb0 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.404+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f704003e440 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.405+0000 7f70457fa640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.405+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f70400ab040 con 0x7f704815d490 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.405+0000 7f704ca50640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.405+0000 7f704ca50640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f702c06ae40 0x7f702c06d300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f703c004840 tx=0x7f703c0078f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.409+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f70400760b0 con 0x7f704815d490 2023-12-16T21:15:15.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.585+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f704815cc20 con 0x7f704815d490 2023-12-16T21:15:15.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.586+0000 7f70457fa640 1 -- 172.21.15.46:0/1242691399 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7040009de0 con 0x7f704815d490 2023-12-16T21:15:15.587 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:15.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f702c06ae40 msgr2=0x7f702c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f702c06ae40 0x7f702c06d300 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f703c004840 tx=0x7f703c0078f0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 msgr2=0x7f704815d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 0x7f704815d940 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7040037040 tx=0x7f7040009750 comp rx=0 tx=0).stop 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 shutdown_connections 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f702c06ae40 0x7f702c06d300 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f704815d490 0x7f704815d940 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7048098cf0 0x7f704815cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 --2- 172.21.15.46:0/1242691399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7048098910 0x7f704806db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 >> 172.21.15.46:0/1242691399 conn(0x7f70480b4720 msgr2=0x7f704815a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 shutdown_connections 2023-12-16T21:15:15.589 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.588+0000 7f704ecb4640 1 -- 172.21.15.46:0/1242691399 wait complete. 2023-12-16T21:15:15.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:15.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.708+0000 7f42fb3d5640 1 Processor -- start 2023-12-16T21:15:15.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.708+0000 7f42fb3d5640 1 -- start start 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42fb3d5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42f406ea50 0x7f42f406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42fb3d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f42f40d0ed0 0x7f42f40d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42fb3d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42f4161d20 con 0x7f42f4160fb0 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42fb3d5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f406f370 con 0x7f42f406ea50 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42fb3d5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f415e760 con 0x7f42f40d0ed0 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9171640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42f406ea50 0x7f42f406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9171640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f42f406ea50 0x7f42f406ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52064/0 (socket says 172.21.15.46:52064) 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9171640 1 -- 172.21.15.46:0/240561003 learned_addr learned my addr 172.21.15.46:0/240561003 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:15.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9972640 1 --2- 172.21.15.46:0/240561003 >> v2:172.21.15.121:3300/0 conn(0x7f42f40d0ed0 0x7f42f40d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9171640 1 -- 172.21.15.46:0/240561003 >> v2:172.21.15.121:3300/0 conn(0x7f42f40d0ed0 msgr2=0x7f42f40d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9171640 1 --2- 172.21.15.46:0/240561003 >> v2:172.21.15.121:3300/0 conn(0x7f42f40d0ed0 0x7f42f40d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.709+0000 7f42f9171640 1 -- 172.21.15.46:0/240561003 >> v1:172.21.15.121:6789/0 conn(0x7f42f4160fb0 legacy=0x7f42f4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.710+0000 7f42f9171640 1 -- 172.21.15.46:0/240561003 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f415d9a0 con 0x7f42f406ea50 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.710+0000 7f42f9171640 1 --2- 172.21.15.46:0/240561003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f406ee30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f42f41407f0 tx=0x7f42ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beb62d3459cdbf7d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.710+0000 7f42f3fff640 1 -- 172.21.15.46:0/240561003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec0317e0 con 0x7f42f406ea50 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.710+0000 7f42f3fff640 1 -- 172.21.15.46:0/240561003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec031940 con 0x7f42f406ea50 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.710+0000 7f42f3fff640 1 -- 172.21.15.46:0/240561003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec031cd0 con 0x7f42f406ea50 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 -- 172.21.15.46:0/240561003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 msgr2=0x7f42f406ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/240561003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f406ee30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f42f41407f0 tx=0x7f42ec02b880 comp rx=0 tx=0).stop 2023-12-16T21:15:15.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 -- 172.21.15.46:0/240561003 shutdown_connections 2023-12-16T21:15:15.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/240561003 >> v2:172.21.15.121:3300/0 conn(0x7f42f40d0ed0 0x7f42f40d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/240561003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f406ee30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 -- 172.21.15.46:0/240561003 >> 172.21.15.46:0/240561003 conn(0x7f42f406a6b0 msgr2=0x7f42f406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:15.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 -- 172.21.15.46:0/240561003 shutdown_connections 2023-12-16T21:15:15.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.711+0000 7f42fb3d5640 1 -- 172.21.15.46:0/240561003 wait complete. 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.712+0000 7f42fb3d5640 1 Processor -- start 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.712+0000 7f42fb3d5640 1 -- start start 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42fb3d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f4157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42fb3d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42f40d0ed0 0x7f42f41e8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42fb3d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42f4160fb0 0x7f42f406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42fb3d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f40cab50 con 0x7f42f406ea50 2023-12-16T21:15:15.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42fb3d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f41584e0 con 0x7f42f4160fb0 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42fb3d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f41587c0 con 0x7f42f40d0ed0 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f4157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f4157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52072/0 (socket says 172.21.15.46:52072) 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 -- 172.21.15.46:0/1338574093 learned_addr learned my addr 172.21.15.46:0/1338574093 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 -- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42f40d0ed0 msgr2=0x7f42f41e8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42f40d0ed0 0x7f42f41e8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 -- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42f4160fb0 msgr2=0x7f42f406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42f4160fb0 0x7f42f406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 -- 172.21.15.46:0/1338574093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec0094d0 con 0x7f42f406ea50 2023-12-16T21:15:15.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.713+0000 7f42f9171640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f4157ec0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f42ec02b8c0 tx=0x7f42ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.714+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec032880 con 0x7f42f406ea50 2023-12-16T21:15:15.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.714+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42ec0329e0 con 0x7f42f406ea50 2023-12-16T21:15:15.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.714+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f4158aa0 con 0x7f42f406ea50 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.714+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec032d90 con 0x7f42f406ea50 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.714+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f41e9340 con 0x7f42f406ea50 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.715+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f40cac90 con 0x7f42f406ea50 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.715+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f42ec0453f0 con 0x7f42f406ea50 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.716+0000 7f42f1ffb640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42e006ae40 0x7f42e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.716+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f42ec0ab9c0 con 0x7f42f406ea50 2023-12-16T21:15:15.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.716+0000 7f42f8970640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42e006ae40 0x7f42e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:15.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.717+0000 7f42f8970640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42e006ae40 0x7f42e006d300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f42f41e8900 tx=0x7f42e80079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:15.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.720+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f42f40cac90 con 0x7f42f406ea50 2023-12-16T21:15:15.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.897+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42f40cac90 con 0x7f42f406ea50 2023-12-16T21:15:15.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.898+0000 7f42f1ffb640 1 -- 172.21.15.46:0/1338574093 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v37) v1 ==== 104+0+11 (secure 0 0 0) 0x7f42ec076840 con 0x7f42f406ea50 2023-12-16T21:15:15.898 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42e006ae40 msgr2=0x7f42e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42e006ae40 0x7f42e006d300 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f42f41e8900 tx=0x7f42e80079b0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 msgr2=0x7f42f4157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f4157ec0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f42ec02b8c0 tx=0x7f42ec0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 shutdown_connections 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42e006ae40 0x7f42e006d300 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42f4160fb0 0x7f42f406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42f40d0ed0 0x7f42f41e8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.899+0000 7f42fb3d5640 1 --2- 172.21.15.46:0/1338574093 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f406ea50 0x7f42f4157ec0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.900+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 >> 172.21.15.46:0/1338574093 conn(0x7f42f406a6b0 msgr2=0x7f42f406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.900+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 shutdown_connections 2023-12-16T21:15:15.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:15.900+0000 7f42fb3d5640 1 -- 172.21.15.46:0/1338574093 wait complete. 2023-12-16T21:15:15.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.021+0000 7f360fcf3640 1 Processor -- start 2023-12-16T21:15:16.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.022+0000 7f360fcf3640 1 -- start start 2023-12-16T21:15:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.022+0000 7f360fcf3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36080d2910 0x7f36080d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.022+0000 7f360fcf3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36080b2b10 con 0x7f36080d3730 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.022+0000 7f360fcf3640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36080d1590 con 0x7f360815d330 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.022+0000 7f360fcf3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36080d3bc0 con 0x7f36080d2910 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.022+0000 7f360d28e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36080d2910 0x7f36080d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.023+0000 7f360e290640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36080d3730 0x7f36080b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36888/0 (socket says 172.21.15.46:36888) 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.023+0000 7f360e290640 1 -- 172.21.15.46:0/146554553 learned_addr learned my addr 172.21.15.46:0/146554553 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.023+0000 7f360d28e640 1 -- 172.21.15.46:0/146554553 >> v1:172.21.15.121:6789/0 conn(0x7f360815d330 legacy=0x7f36080b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.023+0000 7f360d28e640 1 -- 172.21.15.46:0/146554553 >> v1:172.21.15.46:6789/0 conn(0x7f36080d3730 legacy=0x7f36080b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.023+0000 7f360d28e640 1 -- 172.21.15.46:0/146554553 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36080d1870 con 0x7f36080d2910 2023-12-16T21:15:16.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360d28e640 1 --2- 172.21.15.46:0/146554553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f36080d31f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f35f80097a0 tx=0x7f35f802b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d3c5c89b6e0e8d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360ca8d640 1 -- 172.21.15.46:0/146554553 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8030710 con 0x7f36080d2910 2023-12-16T21:15:16.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360ca8d640 1 -- 172.21.15.46:0/146554553 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f802e050 con 0x7f36080d2910 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360ca8d640 1 -- 172.21.15.46:0/146554553 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f8030a90 con 0x7f36080d2910 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 -- 172.21.15.46:0/146554553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 msgr2=0x7f36080d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 --2- 172.21.15.46:0/146554553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f36080d31f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f35f80097a0 tx=0x7f35f802b680 comp rx=0 tx=0).stop 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 -- 172.21.15.46:0/146554553 shutdown_connections 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 --2- 172.21.15.46:0/146554553 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f36080d31f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 -- 172.21.15.46:0/146554553 >> 172.21.15.46:0/146554553 conn(0x7f3608072020 msgr2=0x7f3608072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 -- 172.21.15.46:0/146554553 shutdown_connections 2023-12-16T21:15:16.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.024+0000 7f360fcf3640 1 -- 172.21.15.46:0/146554553 wait complete. 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 Processor -- start 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 -- start start 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f3608159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36080d3730 0x7f36080c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360815d330 0x7f36080cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3608159570 con 0x7f36080d2910 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36080ca6a0 con 0x7f36080d3730 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360fcf3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36080ca980 con 0x7f360815d330 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360da8f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f3608159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360da8f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f3608159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52094/0 (socket says 172.21.15.46:52094) 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.026+0000 7f360da8f640 1 -- 172.21.15.46:0/941169446 learned_addr learned my addr 172.21.15.46:0/941169446 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360d28e640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36080d3730 0x7f36080c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360e290640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360815d330 0x7f36080cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360da8f640 1 -- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360815d330 msgr2=0x7f36080cb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360da8f640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360815d330 0x7f36080cb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360da8f640 1 -- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36080d3730 msgr2=0x7f36080c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360da8f640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36080d3730 0x7f36080c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360da8f640 1 -- 172.21.15.46:0/941169446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f8009400 con 0x7f36080d2910 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360d28e640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36080d3730 0x7f36080c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360da8f640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f3608159030 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f360400b3f0 tx=0x7f360400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3604011830 con 0x7f36080d2910 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3604011990 con 0x7f36080d2910 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36080cac60 con 0x7f36080d2910 2023-12-16T21:15:16.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3604011d40 con 0x7f36080d2910 2023-12-16T21:15:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.027+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36080cb220 con 0x7f36080d2910 2023-12-16T21:15:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.028+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3608069400 con 0x7f36080d2910 2023-12-16T21:15:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.029+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f360401a400 con 0x7f36080d2910 2023-12-16T21:15:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.029+0000 7f35fe7fc640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f35ec06ae40 0x7f35ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.029+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f36040895f0 con 0x7f36080d2910 2023-12-16T21:15:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.029+0000 7f360d28e640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f35ec06ae40 0x7f35ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.030+0000 7f360d28e640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f35ec06ae40 0x7f35ec06d300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f35f8034000 tx=0x7f35f8008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.033+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3604054510 con 0x7f36080d2910 2023-12-16T21:15:16.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.208+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f36080b76a0 con 0x7f36080d2910 2023-12-16T21:15:16.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.209+0000 7f35fe7fc640 1 -- 172.21.15.46:0/941169446 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v37) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3604057b30 con 0x7f36080d2910 2023-12-16T21:15:16.209 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:16.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f35ec06ae40 msgr2=0x7f35ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f35ec06ae40 0x7f35ec06d300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f35f8034000 tx=0x7f35f8008030 comp rx=0 tx=0).stop 2023-12-16T21:15:16.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 msgr2=0x7f3608159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f3608159030 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f360400b3f0 tx=0x7f360400b8b0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 shutdown_connections 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f35ec06ae40 0x7f35ec06d300 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360815d330 0x7f36080cb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36080d3730 0x7f36080c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 --2- 172.21.15.46:0/941169446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36080d2910 0x7f3608159030 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 >> 172.21.15.46:0/941169446 conn(0x7f3608072020 msgr2=0x7f3608072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 shutdown_connections 2023-12-16T21:15:16.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.211+0000 7f360fcf3640 1 -- 172.21.15.46:0/941169446 wait complete. 2023-12-16T21:15:16.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-16T21:15:16.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.331+0000 7f9880f89640 1 Processor -- start 2023-12-16T21:15:16.331 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.331+0000 7f9880f89640 1 -- start start 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f9880f89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f987c06ea50 0x7f987c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f9880f89640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f987c160fb0 0x7f987c161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f9880f89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987c161d20 con 0x7f987c0b3ff0 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f9880f89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c0d1ee0 con 0x7f987c06ea50 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f9880f89640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f987c06f370 con 0x7f987c160fb0 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f987c06ea50 0x7f987c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f987c06ea50 0x7f987c06ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52100/0 (socket says 172.21.15.46:52100) 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987ad9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f987c0b3ff0 0x7f987c0b4460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36892/0 (socket says 172.21.15.46:36892) 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 -- 172.21.15.46:0/1474033973 learned_addr learned my addr 172.21.15.46:0/1474033973 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f9879d9b640 1 --2- 172.21.15.46:0/1474033973 >> v2:172.21.15.121:3300/0 conn(0x7f987c160fb0 0x7f987c161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 -- 172.21.15.46:0/1474033973 >> v2:172.21.15.121:3300/0 conn(0x7f987c160fb0 msgr2=0x7f987c161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:16.333 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 --2- 172.21.15.46:0/1474033973 >> v2:172.21.15.121:3300/0 conn(0x7f987c160fb0 0x7f987c161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 -- 172.21.15.46:0/1474033973 >> v1:172.21.15.46:6789/0 conn(0x7f987c0b3ff0 legacy=0x7f987c0b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987a59c640 1 -- 172.21.15.46:0/1474033973 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c15e760 con 0x7f987c06ea50 2023-12-16T21:15:16.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.332+0000 7f987959a640 1 -- 172.21.15.46:0/1474033973 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270160226 0 0) 0x7f987c161d20 con 0x7f987c0b3ff0 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.333+0000 7f987a59c640 1 --2- 172.21.15.46:0/1474033973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c06ea50 0x7f987c06ee30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b5b30 tx=0x7f986c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8831bfb0d37b11c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.333+0000 7f987959a640 1 -- 172.21.15.46:0/1474033973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c031870 con 0x7f987c06ea50 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.333+0000 7f987959a640 1 -- 172.21.15.46:0/1474033973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986c0319d0 con 0x7f987c06ea50 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.333+0000 7f987959a640 1 -- 172.21.15.46:0/1474033973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986c031d60 con 0x7f987c06ea50 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.333+0000 7f9880f89640 1 -- 172.21.15.46:0/1474033973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c06ea50 msgr2=0x7f987c06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.333+0000 7f9880f89640 1 --2- 172.21.15.46:0/1474033973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c06ea50 0x7f987c06ee30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b5b30 tx=0x7f986c02b910 comp rx=0 tx=0).stop 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.334+0000 7f9880f89640 1 -- 172.21.15.46:0/1474033973 shutdown_connections 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.334+0000 7f9880f89640 1 --2- 172.21.15.46:0/1474033973 >> v2:172.21.15.121:3300/0 conn(0x7f987c160fb0 0x7f987c161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.334+0000 7f9880f89640 1 --2- 172.21.15.46:0/1474033973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c06ea50 0x7f987c06ee30 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.334+0000 7f9880f89640 1 -- 172.21.15.46:0/1474033973 >> 172.21.15.46:0/1474033973 conn(0x7f987c06a6b0 msgr2=0x7f987c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.334+0000 7f9880f89640 1 -- 172.21.15.46:0/1474033973 shutdown_connections 2023-12-16T21:15:16.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.334+0000 7f9880f89640 1 -- 172.21.15.46:0/1474033973 wait complete. 2023-12-16T21:15:16.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.335+0000 7f9880f89640 1 Processor -- start 2023-12-16T21:15:16.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.335+0000 7f9880f89640 1 -- start start 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f987c06ea50 0x7f987c06c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f987c0b3ff0 0x7f987c158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 0x7f987c15c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c158e60 con 0x7f987c160fb0 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c157180 con 0x7f987c06ea50 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987c06c8a0 con 0x7f987c0b3ff0 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 0x7f987c15c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 0x7f987c15c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52110/0 (socket says 172.21.15.46:52110) 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 -- 172.21.15.46:0/3305056132 learned_addr learned my addr 172.21.15.46:0/3305056132 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987a59c640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f987c06ea50 0x7f987c06c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 -- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f987c0b3ff0 msgr2=0x7f987c158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f987c0b3ff0 0x7f987c158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 -- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f987c06ea50 msgr2=0x7f987c06c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f987c06ea50 0x7f987c06c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 -- 172.21.15.46:0/3305056132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0094d0 con 0x7f987c160fb0 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f987ad9d640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 0x7f987c15c9b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9874002230 tx=0x7f987400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987400ce30 con 0x7f987c160fb0 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987400a640 con 0x7f987c160fb0 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c06cb80 con 0x7f987c160fb0 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987400a9f0 con 0x7f987c160fb0 2023-12-16T21:15:16.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.336+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c06d140 con 0x7f987c160fb0 2023-12-16T21:15:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.338+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c069400 con 0x7f987c160fb0 2023-12-16T21:15:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.338+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f987400ab50 con 0x7f987c160fb0 2023-12-16T21:15:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.338+0000 7f986affd640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f985c06ae40 0x7f985c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.338+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f9874088fe0 con 0x7f987c160fb0 2023-12-16T21:15:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.339+0000 7f987a59c640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f985c06ae40 0x7f985c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.340 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.339+0000 7f987a59c640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f985c06ae40 0x7f985c06d300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c03c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.342+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9874053ea0 con 0x7f987c160fb0 2023-12-16T21:15:16.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.519+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f987c070d30 con 0x7f987c160fb0 2023-12-16T21:15:16.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.520+0000 7f986affd640 1 -- 172.21.15.46:0/3305056132 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v37) v1 ==== 97+0+11 (secure 0 0 0) 0x7f9874018020 con 0x7f987c160fb0 2023-12-16T21:15:16.521 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f985c06ae40 msgr2=0x7f985c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f985c06ae40 0x7f985c06d300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f986c036000 tx=0x7f986c03c040 comp rx=0 tx=0).stop 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 msgr2=0x7f987c15c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 0x7f987c15c9b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9874002230 tx=0x7f987400cbe0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 shutdown_connections 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f985c06ae40 0x7f985c06d300 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f987c160fb0 0x7f987c15c9b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f987c0b3ff0 0x7f987c158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 --2- 172.21.15.46:0/3305056132 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f987c06ea50 0x7f987c06c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 >> 172.21.15.46:0/3305056132 conn(0x7f987c06a6b0 msgr2=0x7f987c13e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 shutdown_connections 2023-12-16T21:15:16.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.522+0000 7f9880f89640 1 -- 172.21.15.46:0/3305056132 wait complete. 2023-12-16T21:15:16.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:16.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.640+0000 7f06062ec640 1 Processor -- start 2023-12-16T21:15:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.640+0000 7f06062ec640 1 -- start start 2023-12-16T21:15:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f06062ec640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0600144980 0x7f060015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f06062ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06000ceb70 0x7f06000cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f06062ec640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0600071770 con 0x7f06000b4370 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f06062ec640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06000ccdc0 con 0x7f0600144980 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f06062ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000cd0a0 con 0x7f06000ceb70 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05fffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0600144980 0x7f060015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f0604889640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f06000b4370 0x7f0600071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36894/0 (socket says 172.21.15.46:36894) 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f0604889640 1 -- 172.21.15.46:0/1150512138 learned_addr learned my addr 172.21.15.46:0/1150512138 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05fffff640 1 -- 172.21.15.46:0/1150512138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06000ceb70 msgr2=0x7f06000cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05ff7fe640 1 --2- 172.21.15.46:0/1150512138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06000ceb70 0x7f06000cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05feffd640 1 -- 172.21.15.46:0/1150512138 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620998175 0 0) 0x7f0600071770 con 0x7f06000b4370 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05fffff640 1 --2- 172.21.15.46:0/1150512138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06000ceb70 0x7f06000cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05fffff640 1 -- 172.21.15.46:0/1150512138 >> v1:172.21.15.46:6789/0 conn(0x7f06000b4370 legacy=0x7f0600071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.641+0000 7f05fffff640 1 -- 172.21.15.46:0/1150512138 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000cd380 con 0x7f0600144980 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.642+0000 7f05fffff640 1 --2- 172.21.15.46:0/1150512138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0600144980 0x7f060015c7c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f05f00097a0 tx=0x7f05f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95fa8c48907283b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.642+0000 7f05feffd640 1 -- 172.21.15.46:0/1150512138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0031830 con 0x7f0600144980 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.642+0000 7f05feffd640 1 -- 172.21.15.46:0/1150512138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f0031990 con 0x7f0600144980 2023-12-16T21:15:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.642+0000 7f05feffd640 1 -- 172.21.15.46:0/1150512138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f0031d20 con 0x7f0600144980 2023-12-16T21:15:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 -- 172.21.15.46:0/1150512138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0600144980 msgr2=0x7f060015c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 --2- 172.21.15.46:0/1150512138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0600144980 0x7f060015c7c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f05f00097a0 tx=0x7f05f002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 -- 172.21.15.46:0/1150512138 shutdown_connections 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 --2- 172.21.15.46:0/1150512138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06000ceb70 0x7f06000cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 --2- 172.21.15.46:0/1150512138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0600144980 0x7f060015c7c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 -- 172.21.15.46:0/1150512138 >> 172.21.15.46:0/1150512138 conn(0x7f060006a4d0 msgr2=0x7f060006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 -- 172.21.15.46:0/1150512138 shutdown_connections 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.643+0000 7f06062ec640 1 -- 172.21.15.46:0/1150512138 wait complete. 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.644+0000 7f06062ec640 1 Processor -- start 2023-12-16T21:15:16.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.644+0000 7f06062ec640 1 -- start start 2023-12-16T21:15:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f06062ec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06000b4370 0x7f06001f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f06062ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 0x7f06001f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f06062ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0600144980 0x7f0600157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f06062ec640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06000aecd0 con 0x7f06000ceb70 2023-12-16T21:15:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f06062ec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0600156c00 con 0x7f06000b4370 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f06062ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06000aef20 con 0x7f0600144980 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 0x7f06001f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 0x7f06001f5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52118/0 (socket says 172.21.15.46:52118) 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 -- 172.21.15.46:0/2601839410 learned_addr learned my addr 172.21.15.46:0/2601839410 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f0604889640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0600144980 0x7f0600157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05fffff640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06000b4370 0x7f06001f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 -- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0600144980 msgr2=0x7f0600157530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0600144980 0x7f0600157530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 -- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06000b4370 msgr2=0x7f06001f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06000b4370 0x7f06001f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 -- 172.21.15.46:0/2601839410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f0009400 con 0x7f06000ceb70 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f0604889640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0600144980 0x7f0600157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05fffff640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06000b4370 0x7f06001f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05ff7fe640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 0x7f06001f5d10 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c7f0 tx=0x7f05f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.645+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4004220 con 0x7f06000ceb70 2023-12-16T21:15:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.646+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f4004380 con 0x7f06000ceb70 2023-12-16T21:15:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.646+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06000af200 con 0x7f06000ceb70 2023-12-16T21:15:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.646+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4007840 con 0x7f06000ceb70 2023-12-16T21:15:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.646+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06001f6250 con 0x7f06000ceb70 2023-12-16T21:15:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.647+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0600069400 con 0x7f06000ceb70 2023-12-16T21:15:16.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.647+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f05f4007bb0 con 0x7f06000ceb70 2023-12-16T21:15:16.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.647+0000 7f05fcff9640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f05ec06ae40 0x7f05ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.647+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f05f4089700 con 0x7f06000ceb70 2023-12-16T21:15:16.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.648+0000 7f05fffff640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f05ec06ae40 0x7f05ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.648+0000 7f05fffff640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f05ec06ae40 0x7f05ec06d300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036000 tx=0x7f05f003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.652+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f05f4054610 con 0x7f06000ceb70 2023-12-16T21:15:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.865+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0600155a80 con 0x7f06000ceb70 2023-12-16T21:15:16.866 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.866+0000 7f05fcff9640 1 -- 172.21.15.46:0/2601839410 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f05f4007e60 con 0x7f06000ceb70 2023-12-16T21:15:16.866 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:16.867 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:44.132960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:15:16.867 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-12-16T21:15:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f05ec06ae40 msgr2=0x7f05ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f05ec06ae40 0x7f05ec06d300 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f05f0036000 tx=0x7f05f003c040 comp rx=0 tx=0).stop 2023-12-16T21:15:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 msgr2=0x7f06001f5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 0x7f06001f5d10 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c7f0 tx=0x7f05f400ccb0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 shutdown_connections 2023-12-16T21:15:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f05ec06ae40 0x7f05ec06d300 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f06000ceb70 0x7f06001f5d10 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0600144980 0x7f0600157530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 --2- 172.21.15.46:0/2601839410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06000b4370 0x7f06001f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.867+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 >> 172.21.15.46:0/2601839410 conn(0x7f060006a4d0 msgr2=0x7f060015e040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.868+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 shutdown_connections 2023-12-16T21:15:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.868+0000 7f06062ec640 1 -- 172.21.15.46:0/2601839410 wait complete. 2023-12-16T21:15:16.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.985+0000 7f0e19231640 1 Processor -- start 2023-12-16T21:15:16.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.985+0000 7f0e19231640 1 -- start start 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.985+0000 7f0e19231640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0e14140310 0x7f0e141618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.985+0000 7f0e19231640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e141409d0 0x7f0e1406a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e19231640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e141406f0 con 0x7f0e141428d0 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e19231640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e14161e20 con 0x7f0e141409d0 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e19231640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e14143960 con 0x7f0e14140310 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e141409d0 0x7f0e1406a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f0e141409d0 0x7f0e1406a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52120/0 (socket says 172.21.15.46:52120) 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 -- 172.21.15.46:0/129533336 learned_addr learned my addr 172.21.15.46:0/129533336 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e12d9d640 1 --2- 172.21.15.46:0/129533336 >> v2:172.21.15.121:3300/0 conn(0x7f0e14140310 0x7f0e141618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 -- 172.21.15.46:0/129533336 >> v2:172.21.15.121:3300/0 conn(0x7f0e14140310 msgr2=0x7f0e141618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 --2- 172.21.15.46:0/129533336 >> v2:172.21.15.121:3300/0 conn(0x7f0e14140310 0x7f0e141618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 -- 172.21.15.46:0/129533336 >> v1:172.21.15.121:6789/0 conn(0x7f0e141428d0 legacy=0x7f0e140714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.986+0000 7f0e0a59c640 1 -- 172.21.15.46:0/129533336 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e14142de0 con 0x7f0e141409d0 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e0a59c640 1 --2- 172.21.15.46:0/129533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1406a4d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0e14161f60 tx=0x7f0dfc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83780a766e0fba81 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e1259c640 1 -- 172.21.15.46:0/129533336 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc031870 con 0x7f0e141409d0 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e1259c640 1 -- 172.21.15.46:0/129533336 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc0319d0 con 0x7f0e141409d0 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e1259c640 1 -- 172.21.15.46:0/129533336 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc031d60 con 0x7f0e141409d0 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 -- 172.21.15.46:0/129533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 msgr2=0x7f0e1406a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 --2- 172.21.15.46:0/129533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1406a4d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f0e14161f60 tx=0x7f0dfc02b910 comp rx=0 tx=0).stop 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 -- 172.21.15.46:0/129533336 shutdown_connections 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 --2- 172.21.15.46:0/129533336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1406a4d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 --2- 172.21.15.46:0/129533336 >> v2:172.21.15.121:3300/0 conn(0x7f0e14140310 0x7f0e141618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 -- 172.21.15.46:0/129533336 >> 172.21.15.46:0/129533336 conn(0x7f0e140b1e50 msgr2=0x7f0e140b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 -- 172.21.15.46:0/129533336 shutdown_connections 2023-12-16T21:15:16.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.987+0000 7f0e19231640 1 -- 172.21.15.46:0/129533336 wait complete. 2023-12-16T21:15:16.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.989+0000 7f0e19231640 1 Processor -- start 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.989+0000 7f0e19231640 1 -- start start 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e19231640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e14140310 0x7f0e1413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e19231640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1413d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e19231640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0e141428d0 0x7f0e1413bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e19231640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14145a80 con 0x7f0e141409d0 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e19231640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1413d170 con 0x7f0e14140310 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e19231640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1413c560 con 0x7f0e141428d0 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1413d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52132/0 (socket says 172.21.15.46:52132) 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 -- 172.21.15.46:0/1657039097 learned_addr learned my addr 172.21.15.46:0/1657039097 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e12d9d640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e14140310 0x7f0e1413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e1359e640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0e141428d0 0x7f0e1413bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 -- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0e141428d0 msgr2=0x7f0e1413bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0e141428d0 0x7f0e1413bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 -- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e14140310 msgr2=0x7f0e1413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e14140310 0x7f0e1413dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 -- 172.21.15.46:0/1657039097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc0094d0 con 0x7f0e141409d0 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e1359e640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0e141428d0 0x7f0e1413bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0a59c640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1413d600 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc02b950 tx=0x7f0dfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.990+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc03abe0 con 0x7f0e141409d0 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.991+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc03ad60 con 0x7f0e141409d0 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.991+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1413c840 con 0x7f0e141409d0 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.991+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc044630 con 0x7f0e141409d0 2023-12-16T21:15:16.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.991+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1413ce00 con 0x7f0e141409d0 2023-12-16T21:15:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.992+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e14069400 con 0x7f0e141409d0 2023-12-16T21:15:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.992+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0dfc044790 con 0x7f0e141409d0 2023-12-16T21:15:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.992+0000 7f0e0bfff640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0df406ae40 0x7f0df406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.993+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f0dfc02d040 con 0x7f0e141409d0 2023-12-16T21:15:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.993+0000 7f0e12d9d640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0df406ae40 0x7f0df406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:16.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.993+0000 7f0e12d9d640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0df406ae40 0x7f0df406d300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009aa0 tx=0x7f0e04009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:16.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:16.997+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0dfc076c60 con 0x7f0e141409d0 2023-12-16T21:15:17.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.173+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e1406ea60 con 0x7f0e141409d0 2023-12-16T21:15:17.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.174+0000 7f0e0bfff640 1 -- 172.21.15.46:0/1657039097 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7f0dfc035090 con 0x7f0e141409d0 2023-12-16T21:15:17.175 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:17.175 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:17.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0df406ae40 msgr2=0x7f0df406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0df406ae40 0x7f0df406d300 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009aa0 tx=0x7f0e04009190 comp rx=0 tx=0).stop 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 msgr2=0x7f0e1413d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1413d600 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc02b950 tx=0x7f0dfc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 shutdown_connections 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0df406ae40 0x7f0df406d300 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0e141409d0 0x7f0e1413d600 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0e141428d0 0x7f0e1413bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 --2- 172.21.15.46:0/1657039097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0e14140310 0x7f0e1413dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 >> 172.21.15.46:0/1657039097 conn(0x7f0e140b1e50 msgr2=0x7f0e14161670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 shutdown_connections 2023-12-16T21:15:17.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.176+0000 7f0e19231640 1 -- 172.21.15.46:0/1657039097 wait complete. 2023-12-16T21:15:17.188 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-16T21:15:17.188 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:17.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.293+0000 7f2d52be5640 1 Processor -- start 2023-12-16T21:15:17.294 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.293+0000 7f2d52be5640 1 -- start start 2023-12-16T21:15:17.295 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d52be5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d4c15d330 0x7f2d4c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d52be5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d2910 0x7f2d4c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d52be5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d4c0b2b10 con 0x7f2d4c0d3730 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d52be5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d4c0d3b50 con 0x7f2d4c15d330 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d52be5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c0d1590 con 0x7f2d4c0d2910 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d50981640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d4c15d330 0x7f2d4c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d51182640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2d4c0d3730 0x7f2d4c0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36896/0 (socket says 172.21.15.46:36896) 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d50981640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2d4c15d330 0x7f2d4c0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52144/0 (socket says 172.21.15.46:52144) 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.294+0000 7f2d51182640 1 -- 172.21.15.46:0/2139126108 learned_addr learned my addr 172.21.15.46:0/2139126108 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d4bfff640 1 --2- 172.21.15.46:0/2139126108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d2910 0x7f2d4c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d50981640 1 -- 172.21.15.46:0/2139126108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d2910 msgr2=0x7f2d4c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d50981640 1 --2- 172.21.15.46:0/2139126108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d2910 0x7f2d4c0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d50981640 1 -- 172.21.15.46:0/2139126108 >> v1:172.21.15.46:6789/0 conn(0x7f2d4c0d3730 legacy=0x7f2d4c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d4b7fe640 1 -- 172.21.15.46:0/2139126108 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693078682 0 0) 0x7f2d4c0b2b10 con 0x7f2d4c0d3730 2023-12-16T21:15:17.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d50981640 1 -- 172.21.15.46:0/2139126108 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c0d1870 con 0x7f2d4c15d330 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.295+0000 7f2d50981640 1 --2- 172.21.15.46:0/2139126108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0b8b80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009d90 tx=0x7f2d4402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb209e372867a437 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d4b7fe640 1 -- 172.21.15.46:0/2139126108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4402cbc0 con 0x7f2d4c15d330 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d4b7fe640 1 -- 172.21.15.46:0/2139126108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4402cd20 con 0x7f2d4c15d330 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d4b7fe640 1 -- 172.21.15.46:0/2139126108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44034660 con 0x7f2d4c15d330 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 -- 172.21.15.46:0/2139126108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c15d330 msgr2=0x7f2d4c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 --2- 172.21.15.46:0/2139126108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0b8b80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009d90 tx=0x7f2d4402be10 comp rx=0 tx=0).stop 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 -- 172.21.15.46:0/2139126108 shutdown_connections 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 --2- 172.21.15.46:0/2139126108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d2910 0x7f2d4c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 --2- 172.21.15.46:0/2139126108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0b8b80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 -- 172.21.15.46:0/2139126108 >> 172.21.15.46:0/2139126108 conn(0x7f2d4c072020 msgr2=0x7f2d4c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:17.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 -- 172.21.15.46:0/2139126108 shutdown_connections 2023-12-16T21:15:17.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.296+0000 7f2d52be5640 1 -- 172.21.15.46:0/2139126108 wait complete. 2023-12-16T21:15:17.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.297+0000 7f2d52be5640 1 Processor -- start 2023-12-16T21:15:17.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.297+0000 7f2d52be5640 1 -- start start 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d52be5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 0x7f2d4c1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d52be5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d3730 0x7f2d4c0b89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d52be5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d52be5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c0c9b40 con 0x7f2d4c0d2910 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d52be5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c0cae80 con 0x7f2d4c15d330 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d52be5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c0cb160 con 0x7f2d4c0d3730 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d50981640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 0x7f2d4c1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d50981640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 0x7f2d4c1f32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52150/0 (socket says 172.21.15.46:52150) 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.298+0000 7f2d50981640 1 -- 172.21.15.46:0/657204891 learned_addr learned my addr 172.21.15.46:0/657204891 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d4bfff640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d3730 0x7f2d4c0b89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d51182640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d50981640 1 -- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d3730 msgr2=0x7f2d4c0b89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d50981640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d3730 0x7f2d4c0b89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d50981640 1 -- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d4c15d330 msgr2=0x7f2d4c0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d50981640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d50981640 1 -- 172.21.15.46:0/657204891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44009a70 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d50981640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 0x7f2d4c1f32a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2d4402c3a0 tx=0x7f2d44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44040650 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4402ce90 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d440409f0 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4c0cb440 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.299+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4c0cba00 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.300+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4c069400 con 0x7f2d4c0d2910 2023-12-16T21:15:17.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.300+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2d44040b50 con 0x7f2d4c0d2910 2023-12-16T21:15:17.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.301+0000 7f2d497fa640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2c06ae70 0x7f2d2c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.301+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2d440ac620 con 0x7f2d4c0d2910 2023-12-16T21:15:17.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.301+0000 7f2d4bfff640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2c06ae70 0x7f2d2c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.301+0000 7f2d4bfff640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2c06ae70 0x7f2d2c06d330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c006fd0 tx=0x7f2d3c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.305+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2d44077600 con 0x7f2d4c0d2910 2023-12-16T21:15:17.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.516+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d4c06bdf0 con 0x7f2d4c0d2910 2023-12-16T21:15:17.517 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.516+0000 7f2d497fa640 1 -- 172.21.15.46:0/657204891 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4942 (secure 0 0 0) 0x7f2d4407ac20 con 0x7f2d4c0d2910 2023-12-16T21:15:17.517 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:17.517 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:44.132960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:15:17.518 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2c06ae70 msgr2=0x7f2d2c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2c06ae70 0x7f2d2c06d330 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c006fd0 tx=0x7f2d3c008040 comp rx=0 tx=0).stop 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 msgr2=0x7f2d4c1f32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 0x7f2d4c1f32a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f2d4402c3a0 tx=0x7f2d44004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 shutdown_connections 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2d2c06ae70 0x7f2d2c06d330 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d4c15d330 0x7f2d4c0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d4c0d3730 0x7f2d4c0b89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 --2- 172.21.15.46:0/657204891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2d4c0d2910 0x7f2d4c1f32a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 >> 172.21.15.46:0/657204891 conn(0x7f2d4c072020 msgr2=0x7f2d4c0b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 shutdown_connections 2023-12-16T21:15:17.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.518+0000 7f2d52be5640 1 -- 172.21.15.46:0/657204891 wait complete. 2023-12-16T21:15:17.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:17.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.636+0000 7fc54bac9640 1 Processor -- start 2023-12-16T21:15:17.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.636+0000 7fc54bac9640 1 -- start start 2023-12-16T21:15:17.638 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc54bac9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 0x7fc5440b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc54bac9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc544146840 0x7fc54406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc54bac9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc544146fd0 con 0x7fc544146460 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc54bac9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54406c180 con 0x7fc544146840 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc54bac9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc54406c460 con 0x7fc54406f1c0 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc54a066640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc544146840 0x7fc54406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc549064640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 0x7fc5440b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc549064640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 0x7fc5440b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:39240/0 (socket says 172.21.15.46:39240) 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.637+0000 7fc549064640 1 -- 172.21.15.46:0/1800232101 learned_addr learned my addr 172.21.15.46:0/1800232101 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc548863640 1 -- 172.21.15.46:0/1800232101 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896753417 0 0) 0x7fc544146fd0 con 0x7fc544146460 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc548863640 1 -- 172.21.15.46:0/1800232101 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52c003520 con 0x7fc544146460 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc549064640 1 -- 172.21.15.46:0/1800232101 >> v2:172.21.15.121:3300/0 conn(0x7fc544146840 msgr2=0x7fc54406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc549064640 1 --2- 172.21.15.46:0/1800232101 >> v2:172.21.15.121:3300/0 conn(0x7fc544146840 0x7fc54406baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc549064640 1 -- 172.21.15.46:0/1800232101 >> v1:172.21.15.121:6789/0 conn(0x7fc544146460 legacy=0x7fc54406eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc549064640 1 -- 172.21.15.46:0/1800232101 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54406c740 con 0x7fc54406f1c0 2023-12-16T21:15:17.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.638+0000 7fc54a066640 1 --2- 172.21.15.46:0/1800232101 >> v2:172.21.15.121:3300/0 conn(0x7fc544146840 0x7fc54406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.639+0000 7fc549064640 1 --2- 172.21.15.46:0/1800232101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 0x7fc5440b49d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc538009870 tx=0x7fc53802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ada8b7f688eb1013 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.639+0000 7fc548863640 1 -- 172.21.15.46:0/1800232101 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc538030840 con 0x7fc54406f1c0 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.639+0000 7fc548863640 1 -- 172.21.15.46:0/1800232101 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5380309a0 con 0x7fc54406f1c0 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.639+0000 7fc548863640 1 -- 172.21.15.46:0/1800232101 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc538030d30 con 0x7fc54406f1c0 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 -- 172.21.15.46:0/1800232101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 msgr2=0x7fc5440b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 --2- 172.21.15.46:0/1800232101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 0x7fc5440b49d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc538009870 tx=0x7fc53802b980 comp rx=0 tx=0).stop 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 -- 172.21.15.46:0/1800232101 shutdown_connections 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 --2- 172.21.15.46:0/1800232101 >> v2:172.21.15.121:3300/0 conn(0x7fc544146840 0x7fc54406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 --2- 172.21.15.46:0/1800232101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc54406f1c0 0x7fc5440b49d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 -- 172.21.15.46:0/1800232101 >> 172.21.15.46:0/1800232101 conn(0x7fc54406a6b0 msgr2=0x7fc54406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:17.641 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 -- 172.21.15.46:0/1800232101 shutdown_connections 2023-12-16T21:15:17.642 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.640+0000 7fc54bac9640 1 -- 172.21.15.46:0/1800232101 wait complete. 2023-12-16T21:15:17.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.641+0000 7fc54bac9640 1 Processor -- start 2023-12-16T21:15:17.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.641+0000 7fc54bac9640 1 -- start start 2023-12-16T21:15:17.644 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54bac9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 0x7fc544140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54bac9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc544146460 0x7fc54413f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54bac9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc544146840 0x7fc54413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54bac9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5440ca370 con 0x7fc54406f1c0 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54bac9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441400e0 con 0x7fc544146840 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54bac9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5440cb380 con 0x7fc544146460 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 0x7fc544140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 0x7fc544140ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52160/0 (socket says 172.21.15.46:52160) 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 -- 172.21.15.46:0/313303642 learned_addr learned my addr 172.21.15.46:0/313303642 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54a066640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc544146840 0x7fc54413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549064640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc544146460 0x7fc54413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 -- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc544146460 msgr2=0x7fc54413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc544146460 0x7fc54413f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 -- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc544146840 msgr2=0x7fc54413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc544146840 0x7fc54413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.645 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 -- 172.21.15.46:0/313303642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc534009a70 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc54a066640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc544146840 0x7fc54413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549064640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc544146460 0x7fc54413f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc549865640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 0x7fc544140ec0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c490 tx=0x7fc53c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.642+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0052d0 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.643+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c005430 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.643+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5380094d0 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.643+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c012870 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.643+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5440ca8a0 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.643+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc544069400 con 0x7fc54406f1c0 2023-12-16T21:15:17.646 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.644+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc53c0129d0 con 0x7fc54406f1c0 2023-12-16T21:15:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.644+0000 7fc5427fc640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc53006ae40 0x7fc53006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.645+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fc53c0893e0 con 0x7fc54406f1c0 2023-12-16T21:15:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.645+0000 7fc549064640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc53006ae40 0x7fc53006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.648 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.645+0000 7fc549064640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc53006ae40 0x7fc53006d300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc538036000 tx=0x7fc53803b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.649 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.649+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc53c057d90 con 0x7fc54406f1c0 2023-12-16T21:15:17.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.861+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc544070460 con 0x7fc54406f1c0 2023-12-16T21:15:17.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.861+0000 7fc5427fc640 1 -- 172.21.15.46:0/313303642 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4942 (secure 0 0 0) 0x7fc53c057920 con 0x7fc54406f1c0 2023-12-16T21:15:17.862 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:17.862 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:14:36.098017+0000","modified":"2023-12-16T21:14:44.132960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5244":{"gid":5244,"name":"a","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.46:6835/3134379513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":3134379513},{"type":"v1","addr":"172.21.15.46:6835","nonce":3134379513}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-16T21:15:17.862 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 14 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc53006ae40 msgr2=0x7fc53006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc53006ae40 0x7fc53006d300 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc538036000 tx=0x7fc53803b040 comp rx=0 tx=0).stop 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 msgr2=0x7fc544140ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 0x7fc544140ec0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00c490 tx=0x7fc53c00c950 comp rx=0 tx=0).stop 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 shutdown_connections 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc53006ae40 0x7fc53006d300 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.864 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc544146840 0x7fc54413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc544146460 0x7fc54413f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 --2- 172.21.15.46:0/313303642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc54406f1c0 0x7fc544140ec0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 >> 172.21.15.46:0/313303642 conn(0x7fc54406a6b0 msgr2=0x7fc54406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 shutdown_connections 2023-12-16T21:15:17.865 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.863+0000 7fc54bac9640 1 -- 172.21.15.46:0/313303642 wait complete. 2023-12-16T21:15:17.875 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.981+0000 7ff02d002640 1 Processor -- start 2023-12-16T21:15:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02d002640 1 -- start start 2023-12-16T21:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02d002640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff02806d6a0 0x7ff02815a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02d002640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0280b2020 con 0x7ff0280ce4f0 2023-12-16T21:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02d002640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0280b2300 con 0x7ff028153a20 2023-12-16T21:15:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02d002640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0280b1e10 con 0x7ff02806d6a0 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff02806d6a0 0x7ff02815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.982+0000 7ff02759e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff0280ce4f0 0x7ff02806c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36904/0 (socket says 172.21.15.46:36904) 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.983+0000 7ff02759e640 1 -- 172.21.15.46:0/62917015 learned_addr learned my addr 172.21.15.46:0/62917015 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.983+0000 7ff02659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff02806d6a0 0x7ff02815a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52168/0 (socket says 172.21.15.46:52168) 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.983+0000 7ff02659c640 1 -- 172.21.15.46:0/62917015 >> v1:172.21.15.121:6789/0 conn(0x7ff028153a20 legacy=0x7ff02806cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.983+0000 7ff02659c640 1 -- 172.21.15.46:0/62917015 >> v1:172.21.15.46:6789/0 conn(0x7ff0280ce4f0 legacy=0x7ff02806c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.983+0000 7ff025d9b640 1 -- 172.21.15.46:0/62917015 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211417986 0 0) 0x7ff0280b2020 con 0x7ff0280ce4f0 2023-12-16T21:15:17.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.983+0000 7ff02659c640 1 -- 172.21.15.46:0/62917015 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0280b25e0 con 0x7ff02806d6a0 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02659c640 1 --2- 172.21.15.46:0/62917015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02806d6a0 0x7ff02815a640 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0097a0 tx=0x7ff01c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f486cb1a28e8d079 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff025d9b640 1 -- 172.21.15.46:0/62917015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c031830 con 0x7ff02806d6a0 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff025d9b640 1 -- 172.21.15.46:0/62917015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c031990 con 0x7ff02806d6a0 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff025d9b640 1 -- 172.21.15.46:0/62917015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c031d20 con 0x7ff02806d6a0 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 -- 172.21.15.46:0/62917015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02806d6a0 msgr2=0x7ff02815a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 --2- 172.21.15.46:0/62917015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02806d6a0 0x7ff02815a640 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0097a0 tx=0x7ff01c02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 -- 172.21.15.46:0/62917015 shutdown_connections 2023-12-16T21:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 --2- 172.21.15.46:0/62917015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff02806d6a0 0x7ff02815a640 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 -- 172.21.15.46:0/62917015 >> 172.21.15.46:0/62917015 conn(0x7ff02806a6b0 msgr2=0x7ff02806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 -- 172.21.15.46:0/62917015 shutdown_connections 2023-12-16T21:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.984+0000 7ff02d002640 1 -- 172.21.15.46:0/62917015 wait complete. 2023-12-16T21:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.986+0000 7ff02d002640 1 Processor -- start 2023-12-16T21:15:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.986+0000 7ff02d002640 1 -- start start 2023-12-16T21:15:17.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.986+0000 7ff02d002640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff02806d6a0 0x7ff028137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.986+0000 7ff02d002640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 0x7ff028149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02d002640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff028153a20 0x7ff02814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02d002640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02814a940 con 0x7ff0280ce4f0 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02d002640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0281f0e00 con 0x7ff028153a20 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02d002640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0281384a0 con 0x7ff02806d6a0 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 0x7ff028149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 0x7ff028149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52176/0 (socket says 172.21.15.46:52176) 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 -- 172.21.15.46:0/3939712338 learned_addr learned my addr 172.21.15.46:0/3939712338 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff026d9d640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff02806d6a0 0x7ff028137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02759e640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff028153a20 0x7ff02814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 -- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff02806d6a0 msgr2=0x7ff028137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff02806d6a0 0x7ff028137f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 -- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff028153a20 msgr2=0x7ff02814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff028153a20 0x7ff02814fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 -- 172.21.15.46:0/3939712338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c009400 con 0x7ff0280ce4f0 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff026d9d640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff02806d6a0 0x7ff028137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02659c640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 0x7ff028149250 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff01c02bee0 tx=0x7ff01c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c03ba70 con 0x7ff0280ce4f0 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c03bbd0 con 0x7ff0280ce4f0 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff028149cc0 con 0x7ff0280ce4f0 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01c0453f0 con 0x7ff0280ce4f0 2023-12-16T21:15:17.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.987+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff02814a280 con 0x7ff0280ce4f0 2023-12-16T21:15:17.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.988+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0280cc480 con 0x7ff0280ce4f0 2023-12-16T21:15:17.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.989+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff01c045550 con 0x7ff0280ce4f0 2023-12-16T21:15:17.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.989+0000 7ff00f7fe640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff01406ae40 0x7ff01406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:17.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.989+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff01c0abbf0 con 0x7ff0280ce4f0 2023-12-16T21:15:17.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.989+0000 7ff026d9d640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff01406ae40 0x7ff01406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:17.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.990+0000 7ff026d9d640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff01406ae40 0x7ff01406d300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff020006fd0 tx=0x7ff020008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:17.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:17.993+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff01c076c30 con 0x7ff0280ce4f0 2023-12-16T21:15:18.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.174+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0280cc480 con 0x7ff0280ce4f0 2023-12-16T21:15:18.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.175+0000 7ff00f7fe640 1 -- 172.21.15.46:0/3939712338 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18348 (secure 0 0 0) 0x7ff01c07a250 con 0x7ff0280ce4f0 2023-12-16T21:15:18.176 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:18.176 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":37,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:14:45.779382+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:14:34.660193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-16T21:14:35.078326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-16T21:14:43.306876+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff01406ae40 msgr2=0x7ff01406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff01406ae40 0x7ff01406d300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff020006fd0 tx=0x7ff020008040 comp rx=0 tx=0).stop 2023-12-16T21:15:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 msgr2=0x7ff028149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 0x7ff028149250 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff01c02bee0 tx=0x7ff01c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.178 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 shutdown_connections 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff01406ae40 0x7ff01406d300 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff0280ce4f0 0x7ff028149250 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff028153a20 0x7ff02814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 --2- 172.21.15.46:0/3939712338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff02806d6a0 0x7ff028137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 >> 172.21.15.46:0/3939712338 conn(0x7ff02806a6b0 msgr2=0x7ff02806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 shutdown_connections 2023-12-16T21:15:18.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.177+0000 7ff02d002640 1 -- 172.21.15.46:0/3939712338 wait complete. 2023-12-16T21:15:18.189 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-16T21:15:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.296+0000 7f9d0a4ee640 1 Processor -- start 2023-12-16T21:15:18.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.296+0000 7f9d0a4ee640 1 -- start start 2023-12-16T21:15:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d0a4ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9d04146aa0 0x7f9d040b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d0a4ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d040b0cf0 0x7f9d040b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d0a4ee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d040b1690 con 0x7f9d040ac8e0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d0a4ee640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d04161bb0 con 0x7f9d04146aa0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d0a4ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d04160fe0 con 0x7f9d040b0cf0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d08a8b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9d040ac8e0 0x7f9d040acd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36910/0 (socket says 172.21.15.46:36910) 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d08a8b640 1 -- 172.21.15.46:0/598899417 learned_addr learned my addr 172.21.15.46:0/598899417 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d037fe640 1 --2- 172.21.15.46:0/598899417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d040b0cf0 0x7f9d040b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d03fff640 1 --2- 172.21.15.46:0/598899417 >> v2:172.21.15.121:3300/0 conn(0x7f9d04146aa0 0x7f9d040b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d02ffd640 1 -- 172.21.15.46:0/598899417 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810034413 0 0) 0x7f9d040b1690 con 0x7f9d040ac8e0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d02ffd640 1 -- 172.21.15.46:0/598899417 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cf0003520 con 0x7f9d040ac8e0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d037fe640 1 -- 172.21.15.46:0/598899417 >> v2:172.21.15.121:3300/0 conn(0x7f9d04146aa0 msgr2=0x7f9d040b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d037fe640 1 --2- 172.21.15.46:0/598899417 >> v2:172.21.15.121:3300/0 conn(0x7f9d04146aa0 0x7f9d040b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.297+0000 7f9d037fe640 1 -- 172.21.15.46:0/598899417 >> v1:172.21.15.46:6789/0 conn(0x7f9d040ac8e0 legacy=0x7f9d040acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.298+0000 7f9d037fe640 1 -- 172.21.15.46:0/598899417 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d04146e80 con 0x7f9d040b0cf0 2023-12-16T21:15:18.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.298+0000 7f9d03fff640 1 --2- 172.21.15.46:0/598899417 >> v2:172.21.15.121:3300/0 conn(0x7f9d04146aa0 0x7f9d040b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.298+0000 7f9d037fe640 1 --2- 172.21.15.46:0/598899417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d040b0cf0 0x7f9d040b1150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40097a0 tx=0x7f9cf402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38260ef7b4d48a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d02ffd640 1 -- 172.21.15.46:0/598899417 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf402e070 con 0x7f9d040b0cf0 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d02ffd640 1 -- 172.21.15.46:0/598899417 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf40315a0 con 0x7f9d040b0cf0 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d02ffd640 1 -- 172.21.15.46:0/598899417 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4031930 con 0x7f9d040b0cf0 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/598899417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d040b0cf0 msgr2=0x7f9d040b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/598899417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d040b0cf0 0x7f9d040b1150 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40097a0 tx=0x7f9cf402b840 comp rx=0 tx=0).stop 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/598899417 shutdown_connections 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/598899417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d040b0cf0 0x7f9d040b1150 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/598899417 >> v2:172.21.15.121:3300/0 conn(0x7f9d04146aa0 0x7f9d040b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/598899417 >> 172.21.15.46:0/598899417 conn(0x7f9d0406a4d0 msgr2=0x7f9d0406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/598899417 shutdown_connections 2023-12-16T21:15:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.299+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/598899417 wait complete. 2023-12-16T21:15:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.300+0000 7f9d0a4ee640 1 Processor -- start 2023-12-16T21:15:18.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 -- start start 2023-12-16T21:15:18.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 0x7f9d041583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d040b0cf0 0x7f9d04156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d04146aa0 0x7f9d04159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040b0470 con 0x7f9d040ac8e0 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d040af5b0 con 0x7f9d040b0cf0 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d0a4ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d041588e0 con 0x7f9d04146aa0 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 0x7f9d041583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d03fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 0x7f9d041583a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52184/0 (socket says 172.21.15.46:52184) 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.301+0000 7f9d03fff640 1 -- 172.21.15.46:0/3030251110 learned_addr learned my addr 172.21.15.46:0/3030251110 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d037fe640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d040b0cf0 0x7f9d04156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d08a8b640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d04146aa0 0x7f9d04159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d03fff640 1 -- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d04146aa0 msgr2=0x7f9d04159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d03fff640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d04146aa0 0x7f9d04159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d03fff640 1 -- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d040b0cf0 msgr2=0x7f9d04156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d03fff640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d040b0cf0 0x7f9d04156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d03fff640 1 -- 172.21.15.46:0/3030251110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec009400 con 0x7f9d040ac8e0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d03fff640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 0x7f9d041583a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00ec10 tx=0x7f9cec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec009760 con 0x7f9d040ac8e0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec00cbd0 con 0x7f9d040ac8e0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf4009400 con 0x7f9d040ac8e0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec013470 con 0x7f9d040ac8e0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.302+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d040af7f0 con 0x7f9d040ac8e0 2023-12-16T21:15:18.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.303+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d040b2d00 con 0x7f9d040ac8e0 2023-12-16T21:15:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.304+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f9cec0135d0 con 0x7f9d040ac8e0 2023-12-16T21:15:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.304+0000 7f9d00ff9640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ce406ae40 0x7f9ce406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.304+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f9cec089dd0 con 0x7f9d040ac8e0 2023-12-16T21:15:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.304+0000 7f9d037fe640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ce406ae40 0x7f9ce406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.305+0000 7f9d037fe640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ce406ae40 0x7f9ce406d300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4036000 tx=0x7f9cf403d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:18.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.308+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9cec054cf0 con 0x7f9d040ac8e0 2023-12-16T21:15:18.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.523+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9d04069400 con 0x7f9d040ac8e0 2023-12-16T21:15:18.524 INFO:tasks.ceph.mon.a.smithi046.stderr:2023-12-16T21:15:18.524+0000 7ff1817f7640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:15:18.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.525+0000 7f9d00ff9640 1 -- 172.21.15.46:0/3030251110 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v15) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9cec058310 con 0x7f9d040ac8e0 2023-12-16T21:15:18.526 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ce406ae40 msgr2=0x7f9ce406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ce406ae40 0x7f9ce406d300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4036000 tx=0x7f9cf403d040 comp rx=0 tx=0).stop 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 msgr2=0x7f9d041583a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 0x7f9d041583a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00ec10 tx=0x7f9cec00c520 comp rx=0 tx=0).stop 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 shutdown_connections 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f9ce406ae40 0x7f9ce406d300 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d04146aa0 0x7f9d04159d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d040b0cf0 0x7f9d04156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 --2- 172.21.15.46:0/3030251110 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9d040ac8e0 0x7f9d041583a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 >> 172.21.15.46:0/3030251110 conn(0x7f9d0406a4d0 msgr2=0x7f9d04145d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:18.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 shutdown_connections 2023-12-16T21:15:18.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.529+0000 7f9d0a4ee640 1 -- 172.21.15.46:0/3030251110 wait complete. 2023-12-16T21:15:18.531 INFO:tasks.ceph.mds.a.smithi046.stderr: -3353> 2023-12-16T21:14:41.724+0000 7fccbfcae640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:15:18.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-16T21:15:18.560 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-12-16T21:15:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.651+0000 7f301e7a9640 1 Processor -- start 2023-12-16T21:15:18.652 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f301e7a9640 1 -- start start 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f301e7a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30180d31c0 0x7f30180d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f301e7a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30180ce640 0x7f30180b6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f301e7a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30180d2e80 con 0x7f30180d19b0 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f301e7a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30180d28a0 con 0x7f30180d31c0 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f301e7a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30180d3ae0 con 0x7f30180ce640 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.652+0000 7f3017fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30180d31c0 0x7f30180d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f3017fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f30180d31c0 0x7f30180d35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52210/0 (socket says 172.21.15.46:52210) 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f3017fff640 1 -- 172.21.15.46:0/4226638709 learned_addr learned my addr 172.21.15.46:0/4226638709 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f301cd46640 1 --2- 172.21.15.46:0/4226638709 >> v2:172.21.15.121:3300/0 conn(0x7f30180ce640 0x7f30180b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f3017fff640 1 -- 172.21.15.46:0/4226638709 >> v2:172.21.15.121:3300/0 conn(0x7f30180ce640 msgr2=0x7f30180b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f3017fff640 1 --2- 172.21.15.46:0/4226638709 >> v2:172.21.15.121:3300/0 conn(0x7f30180ce640 0x7f30180b6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.654 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f3017fff640 1 -- 172.21.15.46:0/4226638709 >> v1:172.21.15.121:6789/0 conn(0x7f30180d19b0 legacy=0x7f30180cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.655 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.653+0000 7f3017fff640 1 -- 172.21.15.46:0/4226638709 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30180cdc50 con 0x7f30180d31c0 2023-12-16T21:15:18.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f3017fff640 1 --2- 172.21.15.46:0/4226638709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d31c0 0x7f30180d35a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f301806bf60 tx=0x7f300802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a2580d11fe7274 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:18.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f3016ffd640 1 -- 172.21.15.46:0/4226638709 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008030860 con 0x7f30180d31c0 2023-12-16T21:15:18.656 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f3016ffd640 1 -- 172.21.15.46:0/4226638709 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30080309c0 con 0x7f30180d31c0 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f3016ffd640 1 -- 172.21.15.46:0/4226638709 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008030d50 con 0x7f30180d31c0 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 -- 172.21.15.46:0/4226638709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d31c0 msgr2=0x7f30180d35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 --2- 172.21.15.46:0/4226638709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d31c0 0x7f30180d35a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f301806bf60 tx=0x7f300802b9a0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 -- 172.21.15.46:0/4226638709 shutdown_connections 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 --2- 172.21.15.46:0/4226638709 >> v2:172.21.15.121:3300/0 conn(0x7f30180ce640 0x7f30180b6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 --2- 172.21.15.46:0/4226638709 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d31c0 0x7f30180d35a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 -- 172.21.15.46:0/4226638709 >> 172.21.15.46:0/4226638709 conn(0x7f3018072020 msgr2=0x7f3018072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 -- 172.21.15.46:0/4226638709 shutdown_connections 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.654+0000 7f301e7a9640 1 -- 172.21.15.46:0/4226638709 wait complete. 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.655+0000 7f301e7a9640 1 Processor -- start 2023-12-16T21:15:18.657 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 -- start start 2023-12-16T21:15:18.659 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30180ce640 0x7f30181e4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 0x7f30181f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30180d31c0 0x7f30180c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30181e3350 con 0x7f30180d19b0 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30181e26d0 con 0x7f30180ce640 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.656+0000 7f301e7a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30181e29b0 con 0x7f30180d31c0 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 0x7f30181f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 0x7f30181f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52218/0 (socket says 172.21.15.46:52218) 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 -- 172.21.15.46:0/3111017514 learned_addr learned my addr 172.21.15.46:0/3111017514 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f3017fff640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30180ce640 0x7f30181e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f301cd46640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30180d31c0 0x7f30180c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 -- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30180d31c0 msgr2=0x7f30180c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30180d31c0 0x7f30180c8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 -- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30180ce640 msgr2=0x7f30181e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:18.660 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30180ce640 0x7f30181e4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 -- 172.21.15.46:0/3111017514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30080094d0 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f30177fe640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 0x7f30181f2410 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f300c00c400 tx=0x7f300c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c0052d0 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300c005430 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30181e2c90 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c003830 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.657+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30181e3770 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.658+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30181e3490 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.659+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f300c003990 con 0x7f30180d19b0 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.659+0000 7f3014ff9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f300406ae40 0x7f300406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:18.661 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.659+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6190+0+0 (secure 0 0 0) 0x7f300c088d40 con 0x7f30180d19b0 2023-12-16T21:15:18.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.659+0000 7f3017fff640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f300406ae40 0x7f300406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:18.662 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.660+0000 7f3017fff640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f300406ae40 0x7f300406d300 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3008036000 tx=0x7f3008008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:18.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.663+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f30181e3490 con 0x7f30180d19b0 2023-12-16T21:15:18.875 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:18.875+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f30181e3490 con 0x7f30180d19b0 2023-12-16T21:15:19.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.560+0000 7f3014ff9640 1 -- 172.21.15.46:0/3111017514 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v16) v1 ==== 104+0+0 (secure 0 0 0) 0x7f30181e3490 con 0x7f30180d19b0 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f300406ae40 msgr2=0x7f300406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f300406ae40 0x7f300406d300 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3008036000 tx=0x7f3008008030 comp rx=0 tx=0).stop 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 msgr2=0x7f30181f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 0x7f30181f2410 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f300c00c400 tx=0x7f300c00c8c0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 shutdown_connections 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f300406ae40 0x7f300406d300 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f30180d19b0 0x7f30181f2410 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30180d31c0 0x7f30180c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 --2- 172.21.15.46:0/3111017514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30180ce640 0x7f30181e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 >> 172.21.15.46:0/3111017514 conn(0x7f3018072020 msgr2=0x7f30180cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:19.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 shutdown_connections 2023-12-16T21:15:19.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.563+0000 7f301e7a9640 1 -- 172.21.15.46:0/3111017514 wait complete. 2023-12-16T21:15:19.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-16T21:15:19.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.736+0000 7fa59a4d3640 1 Processor -- start 2023-12-16T21:15:19.737 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.736+0000 7fa59a4d3640 1 -- start start 2023-12-16T21:15:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa59a4d3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa59a4d3640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5940ceb70 0x7fa5940cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa59a4d3640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa594071770 con 0x7fa5940b4370 2023-12-16T21:15:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa59a4d3640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940ccdc0 con 0x7fa5940ceb70 2023-12-16T21:15:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa59a4d3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940cd0a0 con 0x7fa594144980 2023-12-16T21:15:19.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa5937fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fa5940ceb70 0x7fa5940cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa598a70640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fa5940b4370 0x7fa594071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:36928/0 (socket says 172.21.15.46:36928) 2023-12-16T21:15:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa598a70640 1 -- 172.21.15.46:0/3176880705 learned_addr learned my addr 172.21.15.46:0/3176880705 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa593fff640 1 --2- 172.21.15.46:0/3176880705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa592ffd640 1 -- 172.21.15.46:0/3176880705 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 567825493 0 0) 0x7fa594071770 con 0x7fa5940b4370 2023-12-16T21:15:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa592ffd640 1 -- 172.21.15.46:0/3176880705 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa580003520 con 0x7fa5940b4370 2023-12-16T21:15:19.747 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa5937fe640 1 -- 172.21.15.46:0/3176880705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 msgr2=0x7fa59415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa5937fe640 1 --2- 172.21.15.46:0/3176880705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa5937fe640 1 -- 172.21.15.46:0/3176880705 >> v1:172.21.15.46:6789/0 conn(0x7fa5940b4370 legacy=0x7fa594071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.737+0000 7fa5937fe640 1 -- 172.21.15.46:0/3176880705 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940cd380 con 0x7fa5940ceb70 2023-12-16T21:15:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.738+0000 7fa5937fe640 1 --2- 172.21.15.46:0/3176880705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940ceb70 0x7fa5940cc450 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa5840097a0 tx=0x7fa58402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8e719cf951f88 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.738+0000 7fa592ffd640 1 -- 172.21.15.46:0/3176880705 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584031830 con 0x7fa5940ceb70 2023-12-16T21:15:19.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.738+0000 7fa592ffd640 1 -- 172.21.15.46:0/3176880705 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa584031990 con 0x7fa5940ceb70 2023-12-16T21:15:19.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.738+0000 7fa592ffd640 1 -- 172.21.15.46:0/3176880705 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584031d20 con 0x7fa5940ceb70 2023-12-16T21:15:19.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 -- 172.21.15.46:0/3176880705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940ceb70 msgr2=0x7fa5940cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:19.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/3176880705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940ceb70 0x7fa5940cc450 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa5840097a0 tx=0x7fa58402b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 -- 172.21.15.46:0/3176880705 shutdown_connections 2023-12-16T21:15:19.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/3176880705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940ceb70 0x7fa5940cc450 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/3176880705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 -- 172.21.15.46:0/3176880705 >> 172.21.15.46:0/3176880705 conn(0x7fa59406a4d0 msgr2=0x7fa59406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 -- 172.21.15.46:0/3176880705 shutdown_connections 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.739+0000 7fa59a4d3640 1 -- 172.21.15.46:0/3176880705 wait complete. 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.740+0000 7fa59a4d3640 1 Processor -- start 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.740+0000 7fa59a4d3640 1 -- start start 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa59a4d3640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 0x7fa594154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa59a4d3640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5940ceb70 0x7fa594159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:19.750 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa59a4d3640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:19.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa59a4d3640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5941f08c0 con 0x7fa5940b4370 2023-12-16T21:15:19.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa59a4d3640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa594134500 con 0x7fa5940ceb70 2023-12-16T21:15:19.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa59a4d3640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa594134220 con 0x7fa594144980 2023-12-16T21:15:19.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 0x7fa594154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:19.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 0x7fa594154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52240/0 (socket says 172.21.15.46:52240) 2023-12-16T21:15:19.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 -- 172.21.15.46:0/2358441266 learned_addr learned my addr 172.21.15.46:0/2358441266 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:19.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 -- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 msgr2=0x7fa59415a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:19.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 -- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5940ceb70 msgr2=0x7fa594159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:19.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5940ceb70 0x7fa594159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:19.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 -- 172.21.15.46:0/2358441266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa584009400 con 0x7fa5940b4370 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.741+0000 7fa593fff640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 0x7fa594154b70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00d5f0 tx=0x7fa57c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.742+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c0125f0 con 0x7fa5940b4370 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.742+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c012750 con 0x7fa5940b4370 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.742+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5941f0ad0 con 0x7fa5940b4370 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.742+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c012b00 con 0x7fa5940b4370 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.742+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5941f1090 con 0x7fa5940b4370 2023-12-16T21:15:19.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.743+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5941f1220 con 0x7fa5940b4370 2023-12-16T21:15:19.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.743+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fa57c012c60 con 0x7fa5940b4370 2023-12-16T21:15:19.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.744+0000 7fa590ff9640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa57406ae40 0x7fa57406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:19.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.744+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6190+0+0 (secure 0 0 0) 0x7fa57c089cf0 con 0x7fa5940b4370 2023-12-16T21:15:19.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.744+0000 7fa5937fe640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa57406ae40 0x7fa57406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:19.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.744+0000 7fa5937fe640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa57406ae40 0x7fa57406d300 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa584036000 tx=0x7fa584030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:19.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.748+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fa57c08e070 con 0x7fa5940b4370 2023-12-16T21:15:19.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:19.923+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa594069400 con 0x7fa5940b4370 2023-12-16T21:15:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.569+0000 7fa590ff9640 1 -- 172.21.15.46:0/2358441266 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v39) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa57c054bc0 con 0x7fa5940b4370 2023-12-16T21:15:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2023-12-16T21:15:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.575+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa57406ae40 msgr2=0x7fa57406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.575+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa57406ae40 0x7fa57406d300 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa584036000 tx=0x7fa584030000 comp rx=0 tx=0).stop 2023-12-16T21:15:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.575+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 msgr2=0x7fa594154b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:20.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.575+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 0x7fa594154b70 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa57c00d5f0 tx=0x7fa57c00dab0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 shutdown_connections 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fa57406ae40 0x7fa57406d300 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa594144980 0x7fa59415a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5940ceb70 0x7fa594159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 --2- 172.21.15.46:0/2358441266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5940b4370 0x7fa594154b70 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 >> 172.21.15.46:0/2358441266 conn(0x7fa59406a4d0 msgr2=0x7fa594071b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.577+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 shutdown_connections 2023-12-16T21:15:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.578+0000 7fa59a4d3640 1 -- 172.21.15.46:0/2358441266 wait complete. 2023-12-16T21:15:20.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-16T21:15:20.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.711+0000 7f03df03d640 1 Processor -- start 2023-12-16T21:15:20.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.711+0000 7f03df03d640 1 -- start start 2023-12-16T21:15:20.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03df03d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 0x7f03d80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03df03d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03d815e730 0x7f03d80b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03df03d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d815e2a0 con 0x7f03d815a4e0 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03df03d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d815b520 con 0x7f03d815e730 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03df03d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8143870 con 0x7f03d8143490 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03dd5da640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f03d815a4e0 0x7f03d815a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42146/0 (socket says 172.21.15.46:42146) 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03dd5da640 1 -- 172.21.15.46:0/727910816 learned_addr learned my addr 172.21.15.46:0/727910816 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:20.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928044107 0 0) 0x7f03d815e2a0 con 0x7f03d815a4e0 2023-12-16T21:15:20.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 0x7f03d80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03d7fff640 1 --2- 172.21.15.46:0/727910816 >> v2:172.21.15.121:3300/0 conn(0x7f03d815e730 0x7f03d80b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.712+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c0003520 con 0x7f03d815a4e0 2023-12-16T21:15:20.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1023724622 0 0) 0x7f03c0003520 con 0x7f03d815a4e0 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 msgr2=0x7f03d80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 0x7f03d80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 >> v2:172.21.15.121:3300/0 conn(0x7f03d815e730 msgr2=0x7f03d80b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 --2- 172.21.15.46:0/727910816 >> v2:172.21.15.121:3300/0 conn(0x7f03d815e730 0x7f03d80b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d815b240 con 0x7f03d815a4e0 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f03d00019f0 con 0x7f03d815a4e0 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 >> v1:172.21.15.46:6789/0 conn(0x7f03d815a4e0 legacy=0x7f03d815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c0003970 0x7f03c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 0x7f03c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0004d00 0x7f03c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d815b520 con 0x7f03c0003970 2023-12-16T21:15:20.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8143870 con 0x7f03c0004d00 2023-12-16T21:15:20.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c00099a0 con 0x7f03d8143490 2023-12-16T21:15:20.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d7fff640 1 --2- 172.21.15.46:0/727910816 >> v2:172.21.15.121:3300/0 conn(0x7f03d815e730 0x7f03d80b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:20.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c0003970 0x7f03c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dd5da640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0004d00 0x7f03c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03d7fff640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 0x7f03c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dcdd9640 1 -- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 msgr2=0x7f03c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d8143490 0x7f03c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dcdd9640 1 -- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0004d00 msgr2=0x7f03c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0004d00 0x7f03c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.713+0000 7f03dcdd9640 1 -- 172.21.15.46:0/727910816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0003520 con 0x7f03c0003970 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.714+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c0003970 0x7f03c0003d90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f03c8014f00 tx=0x7f03c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb845a738eb55a1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.714+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8034d60 con 0x7f03c0003970 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.714+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c803c440 con 0x7f03c0003970 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.714+0000 7f03d77fe640 1 -- 172.21.15.46:0/727910816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c803c7d0 con 0x7f03c0003970 2023-12-16T21:15:20.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03df03d640 1 -- 172.21.15.46:0/727910816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c0003970 msgr2=0x7f03c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03df03d640 1 --2- 172.21.15.46:0/727910816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03c0003970 0x7f03c0003d90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f03c8014f00 tx=0x7f03c00057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03dd5da640 1 -- 172.21.15.46:0/727910816 reap_dead start 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03df03d640 1 -- 172.21.15.46:0/727910816 shutdown_connections 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03df03d640 1 -- 172.21.15.46:0/727910816 >> 172.21.15.46:0/727910816 conn(0x7f03d806a4d0 msgr2=0x7f03d806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03df03d640 1 -- 172.21.15.46:0/727910816 shutdown_connections 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.715+0000 7f03df03d640 1 -- 172.21.15.46:0/727910816 wait complete. 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.716+0000 7f03df03d640 1 Processor -- start 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.716+0000 7f03df03d640 1 -- start start 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03df03d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0003970 0x7f03d80ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03df03d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d81595b0 0x7f03d81405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03df03d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 0x7f03d8140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03df03d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d813ed50 con 0x7f03d8140b20 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03df03d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d813efc0 con 0x7f03c0003970 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03df03d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d8148570 con 0x7f03d81595b0 2023-12-16T21:15:20.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 0x7f03d8140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 0x7f03d8140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50708/0 (socket says 172.21.15.46:50708) 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 -- 172.21.15.46:0/1844520179 learned_addr learned my addr 172.21.15.46:0/1844520179 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0003970 0x7f03d80ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03d7fff640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d81595b0 0x7f03d81405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 -- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d81595b0 msgr2=0x7f03d81405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d81595b0 0x7f03d81405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 -- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0003970 msgr2=0x7f03d80ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0003970 0x7f03d80ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 -- 172.21.15.46:0/1844520179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8148850 con 0x7f03d8140b20 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.717+0000 7f03dd5da640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 0x7f03d8140fa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f03d80b2550 tx=0x7f03d0007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.718+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d000ac80 con 0x7f03d8140b20 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.718+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d000ade0 con 0x7f03d8140b20 2023-12-16T21:15:20.729 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.718+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d8148b30 con 0x7f03d8140b20 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.718+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d00135f0 con 0x7f03d8140b20 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.718+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d813f2a0 con 0x7f03d8140b20 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.719+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d8069400 con 0x7f03d8140b20 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.719+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f03d0013750 con 0x7f03d8140b20 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.720+0000 7f03d57fa640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03c406ae70 0x7f03c406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.720+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f03d0089cb0 con 0x7f03d8140b20 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.720+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03c406ae70 0x7f03c406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.720+0000 7f03dcdd9640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03c406ae70 0x7f03c406d330 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f03c803e000 tx=0x7f03c8038000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:20.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.724+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f03d0054cb0 con 0x7f03d8140b20 2023-12-16T21:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:20.903+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f03d80b2820 con 0x7f03d8140b20 2023-12-16T21:15:21.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.570+0000 7f03d57fa640 1 -- 172.21.15.46:0/1844520179 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v40) v1 ==== 169+0+0 (secure 0 0 0) 0x7f03d00582d0 con 0x7f03d8140b20 2023-12-16T21:15:21.571 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2023-12-16T21:15:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.578+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03c406ae70 msgr2=0x7f03c406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.578+0000 7f03df03d640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03c406ae70 0x7f03c406d330 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f03c803e000 tx=0x7f03c8038000 comp rx=0 tx=0).stop 2023-12-16T21:15:21.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.578+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 msgr2=0x7f03d8140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.578+0000 7f03df03d640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 0x7f03d8140fa0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f03d80b2550 tx=0x7f03d0007b20 comp rx=0 tx=0).stop 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 shutdown_connections 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f03c406ae70 0x7f03c406d330 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d8140b20 0x7f03d8140fa0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03d81595b0 0x7f03d81405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 --2- 172.21.15.46:0/1844520179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03c0003970 0x7f03d80ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 >> 172.21.15.46:0/1844520179 conn(0x7f03d806a4d0 msgr2=0x7f03d80cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.579+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 shutdown_connections 2023-12-16T21:15:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.580+0000 7f03df03d640 1 -- 172.21.15.46:0/1844520179 wait complete. 2023-12-16T21:15:21.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2023-12-16T21:15:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.713+0000 7fd159a09640 1 Processor -- start 2023-12-16T21:15:21.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.713+0000 7fd159a09640 1 -- start start 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd159a09640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1540b1400 0x7fd15415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd159a09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15415e2a0 con 0x7fd154143490 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd159a09640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15415b240 con 0x7fd15415e730 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd159a09640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15415b520 con 0x7fd1540b1400 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd152ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd154143490 0x7fd1540b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42152/0 (socket says 172.21.15.46:42152) 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd152ffd640 1 -- 172.21.15.46:0/132303388 learned_addr learned my addr 172.21.15.46:0/132303388 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.714+0000 7fd1527fc640 1 --2- 172.21.15.46:0/132303388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1540b1400 0x7fd15415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303799394 0 0) 0x7fd15415e2a0 con 0x7fd154143490 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd13c003520 con 0x7fd154143490 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349305909 0 0) 0x7fd15415b240 con 0x7fd15415e730 2023-12-16T21:15:21.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15415e2a0 con 0x7fd15415e730 2023-12-16T21:15:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2218312609 0 0) 0x7fd13c003520 con 0x7fd154143490 2023-12-16T21:15:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd1527fc640 1 -- 172.21.15.46:0/132303388 >> v1:172.21.15.121:6789/0 conn(0x7fd15415e730 legacy=0x7fd15415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd1527fc640 1 -- 172.21.15.46:0/132303388 >> v1:172.21.15.46:6789/0 conn(0x7fd154143490 legacy=0x7fd1540b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.715+0000 7fd1527fc640 1 -- 172.21.15.46:0/132303388 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15415dc50 con 0x7fd1540b1400 2023-12-16T21:15:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd1527fc640 1 --2- 172.21.15.46:0/132303388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1540b1400 0x7fd15415a4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd1480097d0 tx=0x7fd14802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e9b218a9b03ef5b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c001670 con 0x7fd1540b1400 2023-12-16T21:15:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14802e050 con 0x7fd1540b1400 2023-12-16T21:15:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd151ffb640 1 -- 172.21.15.46:0/132303388 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148030730 con 0x7fd1540b1400 2023-12-16T21:15:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 -- 172.21.15.46:0/132303388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1540b1400 msgr2=0x7fd15415a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 --2- 172.21.15.46:0/132303388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1540b1400 0x7fd15415a4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd1480097d0 tx=0x7fd14802b800 comp rx=0 tx=0).stop 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 -- 172.21.15.46:0/132303388 shutdown_connections 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 --2- 172.21.15.46:0/132303388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1540b1400 0x7fd15415a4e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 -- 172.21.15.46:0/132303388 >> 172.21.15.46:0/132303388 conn(0x7fd15406a4d0 msgr2=0x7fd15406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 -- 172.21.15.46:0/132303388 shutdown_connections 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.716+0000 7fd159a09640 1 -- 172.21.15.46:0/132303388 wait complete. 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.718+0000 7fd159a09640 1 Processor -- start 2023-12-16T21:15:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.718+0000 7fd159a09640 1 -- start start 2023-12-16T21:15:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.718+0000 7fd159a09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1540b1400 0x7fd1540af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.718+0000 7fd159a09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 0x7fd15413b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.718+0000 7fd159a09640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd15415e730 0x7fd15415c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.718+0000 7fd159a09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15413bf20 con 0x7fd154143490 2023-12-16T21:15:21.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd159a09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15415c8a0 con 0x7fd1540b1400 2023-12-16T21:15:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd159a09640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15415cb80 con 0x7fd15415e730 2023-12-16T21:15:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 0x7fd15413b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 0x7fd15413b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50722/0 (socket says 172.21.15.46:50722) 2023-12-16T21:15:21.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 -- 172.21.15.46:0/776592743 learned_addr learned my addr 172.21.15.46:0/776592743 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd152ffd640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1540b1400 0x7fd1540af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1537fe640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd15415e730 0x7fd15415c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 -- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd15415e730 msgr2=0x7fd15415c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd15415e730 0x7fd15415c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 -- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1540b1400 msgr2=0x7fd1540af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1540b1400 0x7fd1540af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 -- 172.21.15.46:0/776592743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148009400 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd1527fc640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 0x7fd15413b9e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148004230 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1480043b0 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15415ce60 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148004520 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.719+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15415d420 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.720+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd154069400 con 0x7fd154143490 2023-12-16T21:15:21.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.721+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd148039040 con 0x7fd154143490 2023-12-16T21:15:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.721+0000 7fd13bfff640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd14006ae40 0x7fd14006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.721+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5338+0+0 (secure 0 0 0) 0x7fd1480aa9f0 con 0x7fd154143490 2023-12-16T21:15:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.722+0000 7fd152ffd640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd14006ae40 0x7fd14006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:21.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.722+0000 7fd152ffd640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd14006ae40 0x7fd14006d300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd14c000f60 tx=0x7fd14c0016b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:21.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.726+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd148075c30 con 0x7fd154143490 2023-12-16T21:15:21.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:21.904+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd15415ccc0 con 0x7fd154143490 2023-12-16T21:15:22.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.582+0000 7fd13bfff640 1 -- 172.21.15.46:0/776592743 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v41) v1 ==== 160+0+0 (secure 0 0 0) 0x7fd15415ccc0 con 0x7fd154143490 2023-12-16T21:15:22.583 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'data_new' removed 2023-12-16T21:15:22.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.585+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd14006ae40 msgr2=0x7fd14006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.585+0000 7fd159a09640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd14006ae40 0x7fd14006d300 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd14c000f60 tx=0x7fd14c0016b0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 msgr2=0x7fd15413b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 0x7fd15413b9e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 shutdown_connections 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd14006ae40 0x7fd14006d300 secure :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd14c000f60 tx=0x7fd14c0016b0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd154143490 0x7fd15413b9e0 secure :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd15415e730 0x7fd15415c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 --2- 172.21.15.46:0/776592743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1540b1400 0x7fd1540af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 >> 172.21.15.46:0/776592743 conn(0x7fd15406a4d0 msgr2=0x7fd1540cdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 shutdown_connections 2023-12-16T21:15:22.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.586+0000 7fd159a09640 1 -- 172.21.15.46:0/776592743 wait complete. 2023-12-16T21:15:22.601 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-16T21:15:22.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.715+0000 7f3e17eec640 1 Processor -- start 2023-12-16T21:15:22.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e17eec640 1 -- start start 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e17eec640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e100d31c0 0x7f3e100d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e17eec640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3e100d19b0 0x7f3e100cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e17eec640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e100d2e80 con 0x7f3e100d1e20 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e17eec640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e100d28a0 con 0x7f3e100d31c0 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e17eec640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e100d3ae0 con 0x7f3e100d19b0 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.716+0000 7f3e15c88640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e100d31c0 0x7f3e100d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15c88640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f3e100d31c0 0x7f3e100d35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50726/0 (socket says 172.21.15.46:50726) 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15c88640 1 -- 172.21.15.46:0/3430927884 learned_addr learned my addr 172.21.15.46:0/3430927884 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e16489640 1 --1- 172.21.15.46:0/3430927884 >> v1:172.21.15.46:6789/0 conn(0x7f3e100d1e20 0x7f3e100b6470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42168/0 (socket says 172.21.15.46:42168) 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15487640 1 --2- 172.21.15.46:0/3430927884 >> v2:172.21.15.121:3300/0 conn(0x7f3e100d19b0 0x7f3e100cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15c88640 1 -- 172.21.15.46:0/3430927884 >> v2:172.21.15.121:3300/0 conn(0x7f3e100d19b0 msgr2=0x7f3e100cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15c88640 1 --2- 172.21.15.46:0/3430927884 >> v2:172.21.15.121:3300/0 conn(0x7f3e100d19b0 0x7f3e100cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15c88640 1 -- 172.21.15.46:0/3430927884 >> v1:172.21.15.46:6789/0 conn(0x7f3e100d1e20 legacy=0x7f3e100b6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.717+0000 7f3e15c88640 1 -- 172.21.15.46:0/3430927884 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e100cdc50 con 0x7f3e100d31c0 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e15c88640 1 --2- 172.21.15.46:0/3430927884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d31c0 0x7f3e100d35a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3e1006bf30 tx=0x7f3e0002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bcd70f74a31bca0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e14c86640 1 -- 172.21.15.46:0/3430927884 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0002cb20 con 0x7f3e100d31c0 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e14c86640 1 -- 172.21.15.46:0/3430927884 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0002cc80 con 0x7f3e100d31c0 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e14c86640 1 -- 172.21.15.46:0/3430927884 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e000343f0 con 0x7f3e100d31c0 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 -- 172.21.15.46:0/3430927884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d31c0 msgr2=0x7f3e100d35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3430927884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d31c0 0x7f3e100d35a0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3e1006bf30 tx=0x7f3e0002bf80 comp rx=0 tx=0).stop 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 -- 172.21.15.46:0/3430927884 shutdown_connections 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3430927884 >> v2:172.21.15.121:3300/0 conn(0x7f3e100d19b0 0x7f3e100cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3430927884 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d31c0 0x7f3e100d35a0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 -- 172.21.15.46:0/3430927884 >> 172.21.15.46:0/3430927884 conn(0x7f3e10072020 msgr2=0x7f3e10072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 -- 172.21.15.46:0/3430927884 shutdown_connections 2023-12-16T21:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.718+0000 7f3e17eec640 1 -- 172.21.15.46:0/3430927884 wait complete. 2023-12-16T21:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 Processor -- start 2023-12-16T21:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 -- start start 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 0x7f3e100b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e100d1e20 0x7f3e101501f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e100d31c0 0x7f3e100c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100c8720 con 0x7f3e100d19b0 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e10159330 con 0x7f3e100d1e20 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e17eec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e10150730 con 0x7f3e100d31c0 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e15c88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 0x7f3e100b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e15c88640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 0x7f3e100b3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50730/0 (socket says 172.21.15.46:50730) 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.720+0000 7f3e15c88640 1 -- 172.21.15.46:0/3616602237 learned_addr learned my addr 172.21.15.46:0/3616602237 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15487640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e100d1e20 0x7f3e101501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e16489640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e100d31c0 0x7f3e100c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15c88640 1 -- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e100d31c0 msgr2=0x7f3e100c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15c88640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e100d31c0 0x7f3e100c8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15c88640 1 -- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e100d1e20 msgr2=0x7f3e101501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15c88640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e100d1e20 0x7f3e101501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15c88640 1 -- 172.21.15.46:0/3616602237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00009b30 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e15c88640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 0x7f3e100b3a10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3e00000f80 tx=0x7f3e00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00009510 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e0003e450 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e10158d70 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0003e800 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.721+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e100c83b0 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.722+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e10069400 con 0x7f3e100d19b0 2023-12-16T21:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.723+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3e0003e960 con 0x7f3e100d19b0 2023-12-16T21:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.723+0000 7f3e0e7fc640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3dfc06ae70 0x7f3dfc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.723+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3e000abd40 con 0x7f3e100d19b0 2023-12-16T21:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.723+0000 7f3e15487640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3dfc06ae70 0x7f3dfc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.724+0000 7f3e15487640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3dfc06ae70 0x7f3dfc06d330 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3e101f62b0 tx=0x7f3e04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:22.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.727+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3e00077230 con 0x7f3e100d19b0 2023-12-16T21:15:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.903+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3e101562e0 con 0x7f3e100d19b0 2023-12-16T21:15:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.906+0000 7f3e0e7fc640 1 -- 172.21.15.46:0/3616602237 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3e0007a850 con 0x7f3e100d19b0 2023-12-16T21:15:22.908 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3dfc06ae70 msgr2=0x7f3dfc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3dfc06ae70 0x7f3dfc06d330 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3e101f62b0 tx=0x7f3e04008040 comp rx=0 tx=0).stop 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 msgr2=0x7f3e100b3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 0x7f3e100b3a10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f3e00000f80 tx=0x7f3e00004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 shutdown_connections 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3dfc06ae70 0x7f3dfc06d330 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e100d31c0 0x7f3e100c8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e100d1e20 0x7f3e101501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 --2- 172.21.15.46:0/3616602237 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3e100d19b0 0x7f3e100b3a10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 >> 172.21.15.46:0/3616602237 conn(0x7f3e10072020 msgr2=0x7f3e100cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 shutdown_connections 2023-12-16T21:15:22.909 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:22.908+0000 7f3e17eec640 1 -- 172.21.15.46:0/3616602237 wait complete. 2023-12-16T21:15:22.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2023-12-16T21:15:23.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.028+0000 7f35839b0640 1 Processor -- start 2023-12-16T21:15:23.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.028+0000 7f35839b0640 1 -- start start 2023-12-16T21:15:23.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f357c160f70 0x7f357c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c0ce480 0x7f357c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357c1614e0 con 0x7f357c15d2d0 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f357c1617c0 con 0x7f357c160f70 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357c161aa0 con 0x7f357c0ce480 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f357c160f70 0x7f357c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f358174c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f357c15d2d0 0x7f357c0b8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42182/0 (socket says 172.21.15.46:42182) 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f357c160f70 0x7f357c0cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50734/0 (socket says 172.21.15.46:50734) 2023-12-16T21:15:23.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f358174c640 1 -- 172.21.15.46:0/1809673263 learned_addr learned my addr 172.21.15.46:0/1809673263 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 -- 172.21.15.46:0/1809673263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c0ce480 msgr2=0x7f357c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 --2- 172.21.15.46:0/1809673263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c0ce480 0x7f357c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 -- 172.21.15.46:0/1809673263 >> v1:172.21.15.46:6789/0 conn(0x7f357c15d2d0 legacy=0x7f357c0b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 -- 172.21.15.46:0/1809673263 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c161d80 con 0x7f357c160f70 2023-12-16T21:15:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3573fff640 1 -- 172.21.15.46:0/1809673263 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914195448 0 0) 0x7f357c1614e0 con 0x7f357c15d2d0 2023-12-16T21:15:23.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3580f4b640 1 --2- 172.21.15.46:0/1809673263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c160f70 0x7f357c0cdf40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f356c0097a0 tx=0x7f356c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68be8888c6f304a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:23.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3573fff640 1 -- 172.21.15.46:0/1809673263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c031830 con 0x7f357c160f70 2023-12-16T21:15:23.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3573fff640 1 -- 172.21.15.46:0/1809673263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c031990 con 0x7f357c160f70 2023-12-16T21:15:23.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f3573fff640 1 -- 172.21.15.46:0/1809673263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c031d20 con 0x7f357c160f70 2023-12-16T21:15:23.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 -- 172.21.15.46:0/1809673263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c160f70 msgr2=0x7f357c0cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.029+0000 7f35839b0640 1 --2- 172.21.15.46:0/1809673263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c160f70 0x7f357c0cdf40 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f356c0097a0 tx=0x7f356c02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.030+0000 7f35839b0640 1 -- 172.21.15.46:0/1809673263 shutdown_connections 2023-12-16T21:15:23.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.030+0000 7f35839b0640 1 --2- 172.21.15.46:0/1809673263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c0ce480 0x7f357c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.030+0000 7f35839b0640 1 --2- 172.21.15.46:0/1809673263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c160f70 0x7f357c0cdf40 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.030+0000 7f35839b0640 1 -- 172.21.15.46:0/1809673263 >> 172.21.15.46:0/1809673263 conn(0x7f357c072230 msgr2=0x7f357c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:23.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.030+0000 7f35839b0640 1 -- 172.21.15.46:0/1809673263 shutdown_connections 2023-12-16T21:15:23.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.030+0000 7f35839b0640 1 -- 172.21.15.46:0/1809673263 wait complete. 2023-12-16T21:15:23.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.031+0000 7f35839b0640 1 Processor -- start 2023-12-16T21:15:23.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.031+0000 7f35839b0640 1 -- start start 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f35839b0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 0x7f357c144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f35839b0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f357c15d2d0 0x7f357c1f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f35839b0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c160f70 0x7f357c0b5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f35839b0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c0c9920 con 0x7f357c0ce480 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f35839b0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c0cd4a0 con 0x7f357c15d2d0 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f35839b0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357c0cd010 con 0x7f357c160f70 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 0x7f357c144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 0x7f357c144b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50736/0 (socket says 172.21.15.46:50736) 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 -- 172.21.15.46:0/3008192691 learned_addr learned my addr 172.21.15.46:0/3008192691 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f3580f4b640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f357c15d2d0 0x7f357c1f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f3581f4d640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c160f70 0x7f357c0b5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 -- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c160f70 msgr2=0x7f357c0b5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c160f70 0x7f357c0b5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 -- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f357c15d2d0 msgr2=0x7f357c1f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f357c15d2d0 0x7f357c1f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 -- 172.21.15.46:0/3008192691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c009400 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f3580f4b640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f357c15d2d0 0x7f357c1f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f358174c640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 0x7f357c144b00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f357800c4d0 tx=0x7f357800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.032+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35780074e0 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.033+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3578007660 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.033+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357c145040 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.033+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578004a20 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.033+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357c0cc6e0 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.034+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357c069400 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.034+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3578004b80 con 0x7f357c0ce480 2023-12-16T21:15:23.040 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.035+0000 7f3571ffb640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f356006ae40 0x7f356006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.035+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3578088850 con 0x7f357c0ce480 2023-12-16T21:15:23.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.035+0000 7f3580f4b640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f356006ae40 0x7f356006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.035+0000 7f3580f4b640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f356006ae40 0x7f356006d300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f356c036000 tx=0x7f356c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:23.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.040+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3578053c00 con 0x7f357c0ce480 2023-12-16T21:15:23.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.280+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) v1 -- 0x7f357c0d0aa0 con 0x7f357c0ce480 2023-12-16T21:15:23.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.585+0000 7f3571ffb640 1 -- 172.21.15.46:0/3008192691 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v124) v1 ==== 147+0+0 (secure 0 0 0) 0x7f3578018020 con 0x7f357c0ce480 2023-12-16T21:15:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f356006ae40 msgr2=0x7f356006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f356006ae40 0x7f356006d300 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f356c036000 tx=0x7f356c008030 comp rx=0 tx=0).stop 2023-12-16T21:15:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 msgr2=0x7f357c144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 0x7f357c144b00 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f357800c4d0 tx=0x7f357800c990 comp rx=0 tx=0).stop 2023-12-16T21:15:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 shutdown_connections 2023-12-16T21:15:23.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f356006ae40 0x7f356006d300 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f357c160f70 0x7f357c0b5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f357c15d2d0 0x7f357c1f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 --2- 172.21.15.46:0/3008192691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f357c0ce480 0x7f357c144b00 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 >> 172.21.15.46:0/3008192691 conn(0x7f357c072230 msgr2=0x7f357c0d1af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 shutdown_connections 2023-12-16T21:15:23.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.587+0000 7f35839b0640 1 -- 172.21.15.46:0/3008192691 wait complete. 2023-12-16T21:15:23.598 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2023-12-16T21:15:23.598 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-16T21:15:23.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:23.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.706+0000 7f0aa7f83640 1 Processor -- start 2023-12-16T21:15:23.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.707+0000 7f0aa7f83640 1 -- start start 2023-12-16T21:15:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.707+0000 7f0aa7f83640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 0x7f0aa00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.707+0000 7f0aa7f83640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0aa00d2910 0x7f0aa00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.707+0000 7f0aa7f83640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa00b2b10 con 0x7f0aa00d2d90 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.707+0000 7f0aa7f83640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa00d1590 con 0x7f0aa00d2910 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.707+0000 7f0aa7f83640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00d1870 con 0x7f0aa015d330 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 0x7f0aa00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa551e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0aa00d2910 0x7f0aa00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 0x7f0aa00b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:59398/0 (socket says 172.21.15.46:59398) 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/903607414 learned_addr learned my addr 172.21.15.46:0/903607414 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa6520640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f0aa00d2d90 0x7f0aa00b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:40152/0 (socket says 172.21.15.46:40152) 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/903607414 >> v2:172.21.15.121:3300/0 conn(0x7f0aa00d2910 msgr2=0x7f0aa00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 --2- 172.21.15.46:0/903607414 >> v2:172.21.15.121:3300/0 conn(0x7f0aa00d2910 0x7f0aa00d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/903607414 >> v1:172.21.15.121:6789/0 conn(0x7f0aa00d2d90 legacy=0x7f0aa00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.708+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/903607414 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa00d1b50 con 0x7f0aa015d330 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa5d1f640 1 --2- 172.21.15.46:0/903607414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 0x7f0aa00b8b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009e50 tx=0x7f0a9002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95194638883f9a50 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa4d1d640 1 -- 172.21.15.46:0/903607414 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9002ca70 con 0x7f0aa015d330 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa4d1d640 1 -- 172.21.15.46:0/903607414 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9002cbd0 con 0x7f0aa015d330 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa4d1d640 1 -- 172.21.15.46:0/903607414 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a900343f0 con 0x7f0aa015d330 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa7f83640 1 -- 172.21.15.46:0/903607414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 msgr2=0x7f0aa00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/903607414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 0x7f0aa00b8b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009e50 tx=0x7f0a9002bed0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa7f83640 1 -- 172.21.15.46:0/903607414 shutdown_connections 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/903607414 >> v2:172.21.15.121:3300/0 conn(0x7f0aa00d2910 0x7f0aa00d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.709+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/903607414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa015d330 0x7f0aa00b8b80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.710+0000 7f0aa7f83640 1 -- 172.21.15.46:0/903607414 >> 172.21.15.46:0/903607414 conn(0x7f0aa0072020 msgr2=0x7f0aa0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.710+0000 7f0aa7f83640 1 -- 172.21.15.46:0/903607414 shutdown_connections 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.710+0000 7f0aa7f83640 1 -- 172.21.15.46:0/903607414 wait complete. 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 Processor -- start 2023-12-16T21:15:23.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 -- start start 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa00d2910 0x7f0aa00cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 0x7f0aa00ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aa015d330 0x7f0aa00ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa01f2f80 con 0x7f0aa00d2d90 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00cca10 con 0x7f0aa015d330 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.711+0000 7f0aa7f83640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00cccf0 con 0x7f0aa00d2910 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa551e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 0x7f0aa00ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa00d2910 0x7f0aa00cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa551e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 0x7f0aa00ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50740/0 (socket says 172.21.15.46:50740) 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa551e640 1 -- 172.21.15.46:0/1368457245 learned_addr learned my addr 172.21.15.46:0/1368457245 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa6520640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aa015d330 0x7f0aa00ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.715 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aa015d330 msgr2=0x7f0aa00ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aa015d330 0x7f0aa00ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/1368457245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 msgr2=0x7f0aa00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 0x7f0aa00ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 -- 172.21.15.46:0/1368457245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a90009b30 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa551e640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 0x7f0aa00ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa5d1f640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa00d2910 0x7f0aa00cc390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00d19b0 tx=0x7f0a90004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90009510 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9003e450 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.712+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa00ccfd0 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.713+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9003e7e0 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.713+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa00cd590 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.713+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa015e9d0 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.714+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0a9003e940 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.714+0000 7f0a9e7fc640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0a8c06ae40 0x7f0a8c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.715+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0a900abe60 con 0x7f0aa00d2910 2023-12-16T21:15:23.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.715+0000 7f0aa551e640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0a8c06ae40 0x7f0a8c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:23.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.715+0000 7f0aa551e640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0a8c06ae40 0x7f0a8c06d300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0156920 tx=0x7f0a94009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:23.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.720+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0a900772d0 con 0x7f0aa00d2910 2023-12-16T21:15:23.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.903+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0aa0069400 con 0x7f0aa00d2910 2023-12-16T21:15:23.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.905+0000 7f0a9e7fc640 1 -- 172.21.15.46:0/1368457245 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+12986 (secure 0 0 0) 0x7f0a9007a8f0 con 0x7f0aa00d2910 2023-12-16T21:15:23.904 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:23.904 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":41,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:22.579747+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6835/3134379513":"2023-12-17T21:15:18.525374+0000","172.21.15.46:6833/3134379513":"2023-12-17T21:15:18.525374+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:23.906 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0a8c06ae40 msgr2=0x7f0a8c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0a8c06ae40 0x7f0a8c06d300 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0156920 tx=0x7f0a94009190 comp rx=0 tx=0).stop 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa00d2910 msgr2=0x7f0aa00cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa00d2910 0x7f0aa00cc390 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00d19b0 tx=0x7f0a90004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 shutdown_connections 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0a8c06ae40 0x7f0a8c06d300 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa00d2d90 0x7f0aa00ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aa015d330 0x7f0aa00ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 --2- 172.21.15.46:0/1368457245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aa00d2910 0x7f0aa00cc390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 >> 172.21.15.46:0/1368457245 conn(0x7f0aa0072020 msgr2=0x7f0aa015c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 shutdown_connections 2023-12-16T21:15:23.907 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:23.907+0000 7f0aa7f83640 1 -- 172.21.15.46:0/1368457245 wait complete. 2023-12-16T21:15:23.917 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:23.918 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:24.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.027+0000 7fd3c0753640 1 Processor -- start 2023-12-16T21:15:24.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.027+0000 7fd3c0753640 1 -- start start 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3c0753640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3b8142270 0x7fd3b81426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3c0753640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b8161d20 con 0x7fd3b8160fb0 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3c0753640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b815e760 con 0x7fd3b806ea50 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3c0753640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b80d1ee0 con 0x7fd3b8142270 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3becf0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd3b8142270 0x7fd3b81426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3bdcee640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fd3b8160fb0 0x7fd3b8161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42194/0 (socket says 172.21.15.46:42194) 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3bdcee640 1 -- 172.21.15.46:0/4056162920 learned_addr learned my addr 172.21.15.46:0/4056162920 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3becf0640 1 -- 172.21.15.46:0/4056162920 >> v1:172.21.15.121:6789/0 conn(0x7fd3b806ea50 legacy=0x7fd3b806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3becf0640 1 -- 172.21.15.46:0/4056162920 >> v1:172.21.15.46:6789/0 conn(0x7fd3b8160fb0 legacy=0x7fd3b8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.028+0000 7fd3becf0640 1 -- 172.21.15.46:0/4056162920 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b815d9a0 con 0x7fd3b8142270 2023-12-16T21:15:24.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.029+0000 7fd3becf0640 1 --2- 172.21.15.46:0/4056162920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8142270 0x7fd3b81426c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00097a0 tx=0x7fd3b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9dbd663d85fb33 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.029+0000 7fd3bd4ed640 1 -- 172.21.15.46:0/4056162920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b002e070 con 0x7fd3b8142270 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.029+0000 7fd3bd4ed640 1 -- 172.21.15.46:0/4056162920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b00315a0 con 0x7fd3b8142270 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.029+0000 7fd3bd4ed640 1 -- 172.21.15.46:0/4056162920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0031930 con 0x7fd3b8142270 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 -- 172.21.15.46:0/4056162920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8142270 msgr2=0x7fd3b81426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 --2- 172.21.15.46:0/4056162920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8142270 0x7fd3b81426c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00097a0 tx=0x7fd3b002b840 comp rx=0 tx=0).stop 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 -- 172.21.15.46:0/4056162920 shutdown_connections 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 --2- 172.21.15.46:0/4056162920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8142270 0x7fd3b81426c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 -- 172.21.15.46:0/4056162920 >> 172.21.15.46:0/4056162920 conn(0x7fd3b806a6b0 msgr2=0x7fd3b806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 -- 172.21.15.46:0/4056162920 shutdown_connections 2023-12-16T21:15:24.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.030+0000 7fd3c0753640 1 -- 172.21.15.46:0/4056162920 wait complete. 2023-12-16T21:15:24.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.031+0000 7fd3c0753640 1 Processor -- start 2023-12-16T21:15:24.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.031+0000 7fd3c0753640 1 -- start start 2023-12-16T21:15:24.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3c0753640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3b806ea50 0x7fd3b81f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3c0753640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3b8142270 0x7fd3b81f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3c0753640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 0x7fd3b81f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3c0753640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b81f63e0 con 0x7fd3b8160fb0 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3c0753640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b81f3f70 con 0x7fd3b806ea50 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3c0753640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b81581d0 con 0x7fd3b8142270 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 0x7fd3b81f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 0x7fd3b81f3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50756/0 (socket says 172.21.15.46:50756) 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 -- 172.21.15.46:0/2093545134 learned_addr learned my addr 172.21.15.46:0/2093545134 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3bdcee640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3b8142270 0x7fd3b81f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3be4ef640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3b806ea50 0x7fd3b81f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 -- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3b8142270 msgr2=0x7fd3b81f5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3b8142270 0x7fd3b81f5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 -- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3b806ea50 msgr2=0x7fd3b81f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3b806ea50 0x7fd3b81f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 -- 172.21.15.46:0/2093545134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b0009400 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3bdcee640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3b8142270 0x7fd3b81f5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3be4ef640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3b806ea50 0x7fd3b81f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3becf0640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 0x7fd3b81f3a30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00097a0 tx=0x7fd3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0032840 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.032+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b00329c0 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.033+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b0032d70 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.033+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b81584b0 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.033+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b8158a70 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.033+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b8069400 con 0x7fd3b8160fb0 2023-12-16T21:15:24.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.034+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd3b003b3f0 con 0x7fd3b8160fb0 2023-12-16T21:15:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.034+0000 7fd3aeffd640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39c06ae40 0x7fd39c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.034+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd3b00ab3e0 con 0x7fd3b8160fb0 2023-12-16T21:15:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.034+0000 7fd3be4ef640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39c06ae40 0x7fd39c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.035+0000 7fd3be4ef640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39c06ae40 0x7fd39c06d300 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4007e90 tx=0x7fd3b40073a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:24.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.038+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd3b00766f0 con 0x7fd3b8160fb0 2023-12-16T21:15:24.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.215+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd3b8140980 con 0x7fd3b8160fb0 2023-12-16T21:15:24.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.216+0000 7fd3aeffd640 1 -- 172.21.15.46:0/2093545134 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd3b0079d10 con 0x7fd3b8160fb0 2023-12-16T21:15:24.216 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39c06ae40 msgr2=0x7fd39c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39c06ae40 0x7fd39c06d300 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4007e90 tx=0x7fd3b40073a0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 msgr2=0x7fd3b81f3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 0x7fd3b81f3a30 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00097a0 tx=0x7fd3b00046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 shutdown_connections 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd39c06ae40 0x7fd39c06d300 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3b8160fb0 0x7fd3b81f3a30 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3b8142270 0x7fd3b81f5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 --2- 172.21.15.46:0/2093545134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3b806ea50 0x7fd3b81f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 >> 172.21.15.46:0/2093545134 conn(0x7fd3b806a6b0 msgr2=0x7fd3b8073500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 shutdown_connections 2023-12-16T21:15:24.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.218+0000 7fd3c0753640 1 -- 172.21.15.46:0/2093545134 wait complete. 2023-12-16T21:15:24.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2023-12-16T21:15:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.335+0000 7f4743568640 1 Processor -- start 2023-12-16T21:15:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.335+0000 7f4743568640 1 -- start start 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4743568640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c15d1d0 0x7f473c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4743568640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f473c0ad640 0x7f473c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4743568640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473c0adae0 con 0x7f473c0aeb80 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4743568640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473c0addc0 con 0x7f473c0ad640 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4743568640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0ae0a0 con 0x7f473c15d1d0 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4740b03640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f473c0ad640 0x7f473c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4741b05640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f473c0aeb80 0x7f473c160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42204/0 (socket says 172.21.15.46:42204) 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4741b05640 1 -- 172.21.15.46:0/2128560896 learned_addr learned my addr 172.21.15.46:0/2128560896 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4741304640 1 --2- 172.21.15.46:0/2128560896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c15d1d0 0x7f473c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4740b03640 1 -- 172.21.15.46:0/2128560896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c15d1d0 msgr2=0x7f473c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4740b03640 1 --2- 172.21.15.46:0/2128560896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c15d1d0 0x7f473c06b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4740b03640 1 -- 172.21.15.46:0/2128560896 >> v1:172.21.15.46:6789/0 conn(0x7f473c0aeb80 legacy=0x7f473c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f473bfff640 1 -- 172.21.15.46:0/2128560896 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532675421 0 0) 0x7f473c0adae0 con 0x7f473c0aeb80 2023-12-16T21:15:24.337 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.336+0000 7f4740b03640 1 -- 172.21.15.46:0/2128560896 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473c0ae380 con 0x7f473c0ad640 2023-12-16T21:15:24.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.337+0000 7f4740b03640 1 --2- 172.21.15.46:0/2128560896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c06ba20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f47300097a0 tx=0x7f473002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405b9b3d7aec5f3e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:24.338 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.337+0000 7f473bfff640 1 -- 172.21.15.46:0/2128560896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730031830 con 0x7f473c0ad640 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.337+0000 7f473bfff640 1 -- 172.21.15.46:0/2128560896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4730031990 con 0x7f473c0ad640 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.337+0000 7f473bfff640 1 -- 172.21.15.46:0/2128560896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730031d20 con 0x7f473c0ad640 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 -- 172.21.15.46:0/2128560896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 msgr2=0x7f473c06ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 --2- 172.21.15.46:0/2128560896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c06ba20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f47300097a0 tx=0x7f473002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 -- 172.21.15.46:0/2128560896 shutdown_connections 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 --2- 172.21.15.46:0/2128560896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c06ba20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 --2- 172.21.15.46:0/2128560896 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c15d1d0 0x7f473c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 -- 172.21.15.46:0/2128560896 >> 172.21.15.46:0/2128560896 conn(0x7f473c070e70 msgr2=0x7f473c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 -- 172.21.15.46:0/2128560896 shutdown_connections 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.338+0000 7f4743568640 1 -- 172.21.15.46:0/2128560896 wait complete. 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.339+0000 7f4743568640 1 Processor -- start 2023-12-16T21:15:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.339+0000 7f4743568640 1 -- start start 2023-12-16T21:15:24.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4743568640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4743568640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c0aeb80 0x7f473c158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4743568640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c15d1d0 0x7f473c0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4743568640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c1f7930 con 0x7f473c0ad640 2023-12-16T21:15:24.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4743568640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473c159180 con 0x7f473c15d1d0 2023-12-16T21:15:24.342 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4743568640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473c0cc690 con 0x7f473c0aeb80 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50768/0 (socket says 172.21.15.46:50768) 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 -- 172.21.15.46:0/118872031 learned_addr learned my addr 172.21.15.46:0/118872031 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4740b03640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c0aeb80 0x7f473c158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741b05640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c15d1d0 0x7f473c0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 -- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c0aeb80 msgr2=0x7f473c158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c0aeb80 0x7f473c158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 -- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c15d1d0 msgr2=0x7f473c0cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c15d1d0 0x7f473c0cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.340+0000 7f4741304640 1 -- 172.21.15.46:0/118872031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4730009400 con 0x7f473c0ad640 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.341+0000 7f4741304640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c1562c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f473400d5f0 tx=0x7f473400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.341+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47340125f0 con 0x7f473c0ad640 2023-12-16T21:15:24.343 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.341+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734012750 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.341+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473c0cc970 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.341+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734012b00 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.341+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473c0ccf30 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.342+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473c0cd0f0 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.342+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4734012c60 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.343+0000 7f4739ffb640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f472806ae40 0x7f472806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.343+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4734089810 con 0x7f473c0ad640 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.343+0000 7f4740b03640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f472806ae40 0x7f472806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:24.344 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.343+0000 7f4740b03640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f472806ae40 0x7f472806d300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4730036000 tx=0x7f4730008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:24.346 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.347+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4734054bb0 con 0x7f473c0ad640 2023-12-16T21:15:24.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:24.591+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) v1 -- 0x7f473c0cc7d0 con 0x7f473c0ad640 2023-12-16T21:15:25.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.591+0000 7f4739ffb640 1 -- 172.21.15.46:0/118872031 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v126) v1 ==== 137+0+0 (secure 0 0 0) 0x7f473c0cc7d0 con 0x7f473c0ad640 2023-12-16T21:15:25.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f472806ae40 msgr2=0x7f472806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f472806ae40 0x7f472806d300 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4730036000 tx=0x7f4730008030 comp rx=0 tx=0).stop 2023-12-16T21:15:25.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 msgr2=0x7f473c1562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.594 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c1562c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f473400d5f0 tx=0x7f473400dab0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 shutdown_connections 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f472806ae40 0x7f472806d300 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f473c15d1d0 0x7f473c0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473c0aeb80 0x7f473c158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 --2- 172.21.15.46:0/118872031 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f473c0ad640 0x7f473c1562c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 >> 172.21.15.46:0/118872031 conn(0x7f473c070e70 msgr2=0x7f473c0b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 shutdown_connections 2023-12-16T21:15:25.595 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.594+0000 7f4743568640 1 -- 172.21.15.46:0/118872031 wait complete. 2023-12-16T21:15:25.607 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-16T21:15:25.722 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.723+0000 7fc9e31a5640 1 Processor -- start 2023-12-16T21:15:25.723 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.723+0000 7fc9e31a5640 1 -- start start 2023-12-16T21:15:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9e31a5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc9dc140310 0x7fc9dc1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9e31a5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc1406f0 con 0x7fc9dc1409d0 2023-12-16T21:15:25.725 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9e31a5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc143960 con 0x7fc9dc06abe0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9e31a5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9dc161e20 con 0x7fc9dc140310 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9dbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc9dc1409d0 0x7fc9dc06a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42218/0 (socket says 172.21.15.46:42218) 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9dbfff640 1 -- 172.21.15.46:0/3132948028 learned_addr learned my addr 172.21.15.46:0/3132948028 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89156356 0 0) 0x7fc9dc1406f0 con 0x7fc9dc1409d0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9cc003520 con 0x7fc9dc1409d0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3132948028 >> v2:172.21.15.121:3300/0 conn(0x7fc9dc140310 0x7fc9dc1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572378373 0 0) 0x7fc9cc003520 con 0x7fc9dc1409d0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 >> v2:172.21.15.121:3300/0 conn(0x7fc9dc140310 msgr2=0x7fc9dc1618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 --2- 172.21.15.46:0/3132948028 >> v2:172.21.15.121:3300/0 conn(0x7fc9dc140310 0x7fc9dc1618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 >> v1:172.21.15.121:6789/0 conn(0x7fc9dc06abe0 legacy=0x7fc9dc06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc142de0 con 0x7fc9dc1409d0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fc9c8001670 con 0x7fc9dc1409d0 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 >> v1:172.21.15.46:6789/0 conn(0x7fc9dc1409d0 legacy=0x7fc9dc06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.726 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9cc003970 0x7fc9cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.724+0000 7fc9db7fe640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc0047c0 0x7fc9cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9db7fe640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9cc0052a0 0x7fc9cc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc161e20 con 0x7fc9cc003970 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc00c170 con 0x7fc9cc0047c0 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc00c330 con 0x7fc9cc0052a0 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9cc003970 0x7fc9cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9dbfff640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc0047c0 0x7fc9cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e1742640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9cc0052a0 0x7fc9cc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e0f41640 1 -- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9cc0052a0 msgr2=0x7fc9cc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9cc0052a0 0x7fc9cc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e0f41640 1 -- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc0047c0 msgr2=0x7fc9cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc0047c0 0x7fc9cc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.727 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e1742640 1 -- 172.21.15.46:0/3132948028 reap_dead start 2023-12-16T21:15:25.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.725+0000 7fc9e0f41640 1 -- 172.21.15.46:0/3132948028 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc003520 con 0x7fc9cc003970 2023-12-16T21:15:25.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9cc003970 0x7fc9cc003d90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000ba00 tx=0x7fc9cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6131511b54c3eebe server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:25.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d00325f0 con 0x7fc9cc003970 2023-12-16T21:15:25.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d0032750 con 0x7fc9cc003970 2023-12-16T21:15:25.728 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9db7fe640 1 -- 172.21.15.46:0/3132948028 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d0032ae0 con 0x7fc9cc003970 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3132948028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9cc003970 msgr2=0x7fc9cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9cc003970 0x7fc9cc003d90 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc9d000ba00 tx=0x7fc9cc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3132948028 shutdown_connections 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3132948028 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9cc003970 0x7fc9cc003d90 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3132948028 >> 172.21.15.46:0/3132948028 conn(0x7fc9dc0b1e50 msgr2=0x7fc9dc0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.726+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3132948028 shutdown_connections 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.727+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3132948028 wait complete. 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.728+0000 7fc9e31a5640 1 Processor -- start 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.728+0000 7fc9e31a5640 1 -- start start 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc003970 0x7fc9dc160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9dc154a20 0x7fc9dc15f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 0x7fc9dc15fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc1f2b20 con 0x7fc9dc1543b0 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc1570a0 con 0x7fc9cc003970 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9dc157380 con 0x7fc9dc154a20 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 0x7fc9dc15fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 0x7fc9dc15fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50788/0 (socket says 172.21.15.46:50788) 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 -- 172.21.15.46:0/3829294379 learned_addr learned my addr 172.21.15.46:0/3829294379 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc003970 0x7fc9dc160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9dbfff640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9dc154a20 0x7fc9dc15f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 -- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9dc154a20 msgr2=0x7fc9dc15f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9dc154a20 0x7fc9dc15f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 -- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc003970 msgr2=0x7fc9dc160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc003970 0x7fc9dc160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 -- 172.21.15.46:0/3829294379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc157660 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e1742640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 0x7fc9dc15fa40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc071430 tx=0x7fc9d400fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4003b80 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d4003d00 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.729+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc157940 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.730+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d4005310 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.730+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc1605b0 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.730+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc143b50 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.731+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc9d4005470 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.731+0000 7fc9d97fa640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9c006ae70 0x7fc9c006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.731+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc9d4089120 con 0x7fc9dc1543b0 2023-12-16T21:15:25.734 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.732+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9c006ae70 0x7fc9c006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:25.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.732+0000 7fc9e0f41640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9c006ae70 0x7fc9c006d330 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc004740 tx=0x7fc9d003e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:25.735 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.735+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc9dc143b50 con 0x7fc9dc1543b0 2023-12-16T21:15:25.914 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.914+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc9dc143b50 con 0x7fc9dc1543b0 2023-12-16T21:15:25.917 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.917+0000 7fc9d97fa640 1 -- 172.21.15.46:0/3829294379 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc9d4054580 con 0x7fc9dc1543b0 2023-12-16T21:15:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9c006ae70 msgr2=0x7fc9c006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9c006ae70 0x7fc9c006d330 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc004740 tx=0x7fc9d003e040 comp rx=0 tx=0).stop 2023-12-16T21:15:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 msgr2=0x7fc9dc15fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 0x7fc9dc15fa40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc071430 tx=0x7fc9d400fd40 comp rx=0 tx=0).stop 2023-12-16T21:15:25.919 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 shutdown_connections 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9c006ae70 0x7fc9c006d330 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9dc1543b0 0x7fc9dc15fa40 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9dc154a20 0x7fc9dc15f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 --2- 172.21.15.46:0/3829294379 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9cc003970 0x7fc9dc160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 >> 172.21.15.46:0/3829294379 conn(0x7fc9dc0b1e50 msgr2=0x7fc9dc0b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 shutdown_connections 2023-12-16T21:15:25.920 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:25.919+0000 7fc9e31a5640 1 -- 172.21.15.46:0/3829294379 wait complete. 2023-12-16T21:15:25.930 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-16T21:15:25.930 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:15:25.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:15:26.034 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-16T21:15:26.034 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:26.035 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:26.050 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:15:26.051 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:15:26.051 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:26.068 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:15:26.068 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:26.068 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:26.106 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:15:26.106 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:15:26.106 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:26.125 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:15:26.125 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-16T21:15:26.126 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:26.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.253+0000 7f36fbe1c640 1 Processor -- start 2023-12-16T21:15:26.253 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.253+0000 7f36fbe1c640 1 -- start start 2023-12-16T21:15:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36fbe1c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f36f4143490 0x7f36f40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36fbe1c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36f415abf0 0x7f36f40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36fbe1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36f415e2a0 con 0x7f36f415e730 2023-12-16T21:15:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36fbe1c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f415b520 con 0x7f36f415abf0 2023-12-16T21:15:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36fbe1c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f4143870 con 0x7f36f4143490 2023-12-16T21:15:26.254 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36fa3b9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f36f415abf0 0x7f36f40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36f93b7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f36f415e730 0x7f36f415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42220/0 (socket says 172.21.15.46:42220) 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36f93b7640 1 -- 172.21.15.46:0/4127612264 learned_addr learned my addr 172.21.15.46:0/4127612264 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.254+0000 7f36f9bb8640 1 --2- 172.21.15.46:0/4127612264 >> v2:172.21.15.121:3300/0 conn(0x7f36f4143490 0x7f36f40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.255+0000 7f36fa3b9640 1 -- 172.21.15.46:0/4127612264 >> v2:172.21.15.121:3300/0 conn(0x7f36f4143490 msgr2=0x7f36f40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.255+0000 7f36fa3b9640 1 --2- 172.21.15.46:0/4127612264 >> v2:172.21.15.121:3300/0 conn(0x7f36f4143490 0x7f36f40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.255+0000 7f36fa3b9640 1 -- 172.21.15.46:0/4127612264 >> v1:172.21.15.46:6789/0 conn(0x7f36f415e730 legacy=0x7f36f415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.255+0000 7f36fa3b9640 1 -- 172.21.15.46:0/4127612264 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f415b240 con 0x7f36f415abf0 2023-12-16T21:15:26.255 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.255+0000 7f36f8bb6640 1 -- 172.21.15.46:0/4127612264 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25876572 0 0) 0x7f36f415e2a0 con 0x7f36f415e730 2023-12-16T21:15:26.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.255+0000 7f36fa3b9640 1 --2- 172.21.15.46:0/4127612264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f415abf0 0x7f36f40b1230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f36f406e620 tx=0x7f36e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7098416c75abc97 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:26.256 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36f8bb6640 1 -- 172.21.15.46:0/4127612264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e402cc80 con 0x7f36f415abf0 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36f8bb6640 1 -- 172.21.15.46:0/4127612264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e402cde0 con 0x7f36f415abf0 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36f8bb6640 1 -- 172.21.15.46:0/4127612264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e4034660 con 0x7f36f415abf0 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 -- 172.21.15.46:0/4127612264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f415abf0 msgr2=0x7f36f40b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/4127612264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f415abf0 0x7f36f40b1230 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f36f406e620 tx=0x7f36e402bed0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 -- 172.21.15.46:0/4127612264 shutdown_connections 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/4127612264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f415abf0 0x7f36f40b1230 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/4127612264 >> v2:172.21.15.121:3300/0 conn(0x7f36f4143490 0x7f36f40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 -- 172.21.15.46:0/4127612264 >> 172.21.15.46:0/4127612264 conn(0x7f36f406a4d0 msgr2=0x7f36f406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 -- 172.21.15.46:0/4127612264 shutdown_connections 2023-12-16T21:15:26.257 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.256+0000 7f36fbe1c640 1 -- 172.21.15.46:0/4127612264 wait complete. 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.257+0000 7f36fbe1c640 1 Processor -- start 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.257+0000 7f36fbe1c640 1 -- start start 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fbe1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 0x7f36f4159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fbe1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36f415abf0 0x7f36f413e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fbe1c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36f415e730 0x7f36f406c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fbe1c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f41f1db0 con 0x7f36f4143490 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fbe1c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f41f1230 con 0x7f36f415e730 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fbe1c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f41f1510 con 0x7f36f415abf0 2023-12-16T21:15:26.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36f9bb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 0x7f36f4159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36f9bb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 0x7f36f4159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50806/0 (socket says 172.21.15.46:50806) 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36f9bb8640 1 -- 172.21.15.46:0/1269265867 learned_addr learned my addr 172.21.15.46:0/1269265867 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36fa3b9640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36f415e730 0x7f36f406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.258+0000 7f36f93b7640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36f415abf0 0x7f36f413e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f9bb8640 1 -- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36f415abf0 msgr2=0x7f36f413e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f9bb8640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36f415abf0 0x7f36f413e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f9bb8640 1 -- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36f415e730 msgr2=0x7f36f406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f9bb8640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36f415e730 0x7f36f406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f9bb8640 1 -- 172.21.15.46:0/1269265867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e4009a70 con 0x7f36f4143490 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f9bb8640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 0x7f36f4159650 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00d800 tx=0x7f36ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:26.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec00b6c0 con 0x7f36f4143490 2023-12-16T21:15:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ec00b820 con 0x7f36f4143490 2023-12-16T21:15:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f41f17f0 con 0x7f36f4143490 2023-12-16T21:15:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec00bbb0 con 0x7f36f4143490 2023-12-16T21:15:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.259+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f41f0df0 con 0x7f36f4143490 2023-12-16T21:15:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.260+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f4069400 con 0x7f36f4143490 2023-12-16T21:15:26.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.261+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f36ec00bd10 con 0x7f36f4143490 2023-12-16T21:15:26.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.261+0000 7f36f27fc640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36e006ae40 0x7f36e006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.261+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f36ec089760 con 0x7f36f4143490 2023-12-16T21:15:26.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.261+0000 7f36f93b7640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36e006ae40 0x7f36e006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.262+0000 7f36f93b7640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36e006ae40 0x7f36e006d300 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f36f40cbcc0 tx=0x7f36e8007c00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:26.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.265+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f36ec054ba0 con 0x7f36f4143490 2023-12-16T21:15:26.475 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.476+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36f40b2820 con 0x7f36f4143490 2023-12-16T21:15:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.476+0000 7f36f27fc640 1 -- 172.21.15.46:0/1269265867 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+3698 (secure 0 0 0) 0x7f36ec0581c0 con 0x7f36f4143490 2023-12-16T21:15:26.476 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:26.476 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":16,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4868,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2253488443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2253488443},{"type":"v1","addr":"172.21.15.121:6836","nonce":2253488443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/148457539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":148457539},{"type":"v1","addr":"172.21.15.121:6837","nonce":148457539}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5241,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/1486684639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1486684639},{"type":"v1","addr":"172.21.15.46:6834","nonce":1486684639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5838,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/734089901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":734089901},{"type":"v1","addr":"172.21.15.46:6835","nonce":734089901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":16}],"filesystems":[]} 2023-12-16T21:15:26.476 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 16 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36e006ae40 msgr2=0x7f36e006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36e006ae40 0x7f36e006d300 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f36f40cbcc0 tx=0x7f36e8007c00 comp rx=0 tx=0).stop 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 msgr2=0x7f36f4159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 0x7f36f4159650 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f36ec00d800 tx=0x7f36ec00dcc0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 shutdown_connections 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f36e006ae40 0x7f36e006d300 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36f415e730 0x7f36f406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36f415abf0 0x7f36f413e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 --2- 172.21.15.46:0/1269265867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f36f4143490 0x7f36f4159650 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 >> 172.21.15.46:0/1269265867 conn(0x7f36f406a4d0 msgr2=0x7f36f406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:26.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 shutdown_connections 2023-12-16T21:15:26.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.478+0000 7f36fbe1c640 1 -- 172.21.15.46:0/1269265867 wait complete. 2023-12-16T21:15:26.488 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-16T21:15:26.489 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-16T21:15:26.489 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-16T21:15:26.489 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:26.489 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-16T21:15:26.489 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-16T21:15:26.489 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-16T21:15:26.489 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:26.490 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-16T21:15:26.490 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-16T21:15:26.490 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-16T21:15:26.490 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:26.490 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-16T21:15:26.490 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-16T21:15:26.490 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-16T21:15:26.491 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:26.499 INFO:tasks.ceph.mds.a:Stopped 2023-12-16T21:15:26.499 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-16T21:15:26.500 INFO:tasks.ceph.mds.c:Stopped 2023-12-16T21:15:26.500 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-16T21:15:26.503 INFO:tasks.ceph.mds.b:Stopped 2023-12-16T21:15:26.503 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-16T21:15:26.504 INFO:tasks.ceph.mds.d:Stopped 2023-12-16T21:15:26.504 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-16T21:15:26.542 INFO:tasks.ceph.mds.a:Started 2023-12-16T21:15:26.542 DEBUG:teuthology.parallel:result is None 2023-12-16T21:15:26.542 INFO:tasks.ceph.mds.c:Started 2023-12-16T21:15:26.543 DEBUG:teuthology.parallel:result is None 2023-12-16T21:15:26.545 INFO:tasks.ceph.mds.b:Started 2023-12-16T21:15:26.546 DEBUG:teuthology.parallel:result is None 2023-12-16T21:15:26.546 INFO:tasks.ceph.mds.d:Started 2023-12-16T21:15:26.546 DEBUG:teuthology.parallel:result is None 2023-12-16T21:15:26.547 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-16T21:15:26.624 INFO:tasks.ceph.mds.a.smithi046.stdout:starting mds.a at 2023-12-16T21:15:26.625 INFO:tasks.ceph.mds.c.smithi046.stdout:starting mds.c at 2023-12-16T21:15:26.627 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2023-12-16T21:15:26.628 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-12-16T21:15:26.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.658+0000 7f97bfc1c640 1 Processor -- start 2023-12-16T21:15:26.658 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.658+0000 7f97bfc1c640 1 -- start start 2023-12-16T21:15:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.659+0000 7f97bfc1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 0x7f97b80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.659+0000 7f97bfc1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b80b2b10 con 0x7f97b80d2910 2023-12-16T21:15:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.659+0000 7f97bfc1c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b80d1590 con 0x7f97b815d330 2023-12-16T21:15:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.659+0000 7f97bfc1c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80d1870 con 0x7f97b80d3900 2023-12-16T21:15:26.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.659+0000 7f97bd1b7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f97b80d2910 0x7f97b80d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42226/0 (socket says 172.21.15.46:42226) 2023-12-16T21:15:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.659+0000 7f97bd1b7640 1 -- 172.21.15.46:0/3341343147 learned_addr learned my addr 172.21.15.46:0/3341343147 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97be1b9640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 0x7f97b80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492962564 0 0) 0x7f97b80b2b10 con 0x7f97b80d2910 2023-12-16T21:15:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a4003520 con 0x7f97b80d2910 2023-12-16T21:15:26.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266062141 0 0) 0x7f97a4003520 con 0x7f97b80d2910 2023-12-16T21:15:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 msgr2=0x7f97b80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 0x7f97b80b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 >> v1:172.21.15.121:6789/0 conn(0x7f97b815d330 legacy=0x7f97b80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80d1b50 con 0x7f97b80d2910 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f97b4001670 con 0x7f97b80d2910 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 >> v1:172.21.15.46:6789/0 conn(0x7f97b80d2910 legacy=0x7f97b80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 0x7f97a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a4004d00 0x7f97a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80d1870 con 0x7f97a4003ee0 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a400ba90 con 0x7f97a4004d00 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a400bc50 con 0x7f97b80d3900 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd9b8640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 0x7f97a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97be1b9640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a4004d00 0x7f97a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd1b7640 1 -- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 msgr2=0x7f97a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80d3900 0x7f97a40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd1b7640 1 -- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a4004d00 msgr2=0x7f97a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97a4004d00 0x7f97a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.660+0000 7f97bd1b7640 1 -- 172.21.15.46:0/3341343147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a4003520 con 0x7f97a4003ee0 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.661+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97a40047c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c210 tx=0x7f97a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffb61bdc1afd390 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:26.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.661+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b4005ad0 con 0x7f97a4003ee0 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.661+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b4005c50 con 0x7f97a4003ee0 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.661+0000 7f97bc9b6640 1 -- 172.21.15.46:0/3341343147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b402c8d0 con 0x7f97a4003ee0 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97bfc1c640 1 -- 172.21.15.46:0/3341343147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 msgr2=0x7f97a40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/3341343147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97a40047c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f97b400c210 tx=0x7f97a40057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97be1b9640 1 -- 172.21.15.46:0/3341343147 reap_dead start 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97bfc1c640 1 -- 172.21.15.46:0/3341343147 shutdown_connections 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97bfc1c640 1 -- 172.21.15.46:0/3341343147 >> 172.21.15.46:0/3341343147 conn(0x7f97b8072020 msgr2=0x7f97b8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97bfc1c640 1 -- 172.21.15.46:0/3341343147 shutdown_connections 2023-12-16T21:15:26.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.662+0000 7f97bfc1c640 1 -- 172.21.15.46:0/3341343147 wait complete. 2023-12-16T21:15:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.663+0000 7f97bfc1c640 1 Processor -- start 2023-12-16T21:15:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.663+0000 7f97bfc1c640 1 -- start start 2023-12-16T21:15:26.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bfc1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97b814daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bfc1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80c86b0 0x7f97b80c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bfc1c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b81f3ce0 0x7f97b80cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bfc1c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b814dfe0 con 0x7f97a4003ee0 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bfc1c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80c9200 con 0x7f97b81f3ce0 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bfc1c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80ca330 con 0x7f97b80c86b0 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97b814daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97b814daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50830/0 (socket says 172.21.15.46:50830) 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 -- 172.21.15.46:0/1903283209 learned_addr learned my addr 172.21.15.46:0/1903283209 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:26.682 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97be1b9640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b81f3ce0 0x7f97b80cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80c86b0 0x7f97b80c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 -- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80c86b0 msgr2=0x7f97b80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80c86b0 0x7f97b80c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 -- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b81f3ce0 msgr2=0x7f97b80cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b81f3ce0 0x7f97b80cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 -- 172.21.15.46:0/1903283209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80ca610 con 0x7f97a4003ee0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97be1b9640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b81f3ce0 0x7f97b80cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97bd9b8640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97b814daa0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f97a800f7c0 tx=0x7f97a800fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.664+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a800fed0 con 0x7f97a4003ee0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.665+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a80052e0 con 0x7f97a4003ee0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.665+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b80ca8f0 con 0x7f97a4003ee0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.665+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a800d970 con 0x7f97a4003ee0 2023-12-16T21:15:26.683 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.665+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b80caeb0 con 0x7f97a4003ee0 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.666+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b8069400 con 0x7f97a4003ee0 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.666+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f97a800dad0 con 0x7f97a4003ee0 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.666+0000 7f97ae7fc640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f979c06ae40 0x7f979c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.667+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f97a8089400 con 0x7f97a4003ee0 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.667+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f979c06ae40 0x7f979c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.667+0000 7f97bd1b7640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f979c06ae40 0x7f979c06d300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f97b4034000 tx=0x7f97b4009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:26.684 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.671+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f97a80547b0 con 0x7f97a4003ee0 2023-12-16T21:15:26.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2023-12-16T21:15:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:26.860+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f97b815cb50 con 0x7f97a4003ee0 2023-12-16T21:15:27.837 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:15:27.841 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:15:27.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.930+0000 7f97ae7fc640 1 -- 172.21.15.46:0/1903283209 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v46) v1 ==== 97+0+0 (secure 0 0 0) 0x7f97a8057dd0 con 0x7f97a4003ee0 2023-12-16T21:15:27.930 INFO:teuthology.orchestra.run.smithi046.stderr: removed all blocklist entries 2023-12-16T21:15:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f979c06ae40 msgr2=0x7f979c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f979c06ae40 0x7f979c06d300 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f97b4034000 tx=0x7f97b4009030 comp rx=0 tx=0).stop 2023-12-16T21:15:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 msgr2=0x7f97b814daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97b814daa0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f97a800f7c0 tx=0x7f97a800fc80 comp rx=0 tx=0).stop 2023-12-16T21:15:27.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 shutdown_connections 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f979c06ae40 0x7f979c06d300 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97b81f3ce0 0x7f97b80cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97b80c86b0 0x7f97b80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 --2- 172.21.15.46:0/1903283209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f97a4003ee0 0x7f97b814daa0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 >> 172.21.15.46:0/1903283209 conn(0x7f97b8072020 msgr2=0x7f97b80b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 shutdown_connections 2023-12-16T21:15:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:27.933+0000 7f97bfc1c640 1 -- 172.21.15.46:0/1903283209 wait complete. 2023-12-16T21:15:27.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-16T21:15:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.102+0000 7f49f8305640 1 Processor -- start 2023-12-16T21:15:28.102 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.102+0000 7f49f8305640 1 -- start start 2023-12-16T21:15:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f8305640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 0x7f49f01618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f8305640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f49f006abe0 0x7f49f006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f8305640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49f01406f0 con 0x7f49f01409d0 2023-12-16T21:15:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f8305640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49f0161e20 con 0x7f49f006abe0 2023-12-16T21:15:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f8305640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f0143960 con 0x7f49f0140310 2023-12-16T21:15:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f58a0640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f49f01409d0 0x7f49f006a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42232/0 (socket says 172.21.15.46:42232) 2023-12-16T21:15:28.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f58a0640 1 -- 172.21.15.46:0/1297839387 learned_addr learned my addr 172.21.15.46:0/1297839387 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f60a1640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 0x7f49f01618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f68a2640 1 --2- 172.21.15.46:0/1297839387 >> v2:172.21.15.121:3300/0 conn(0x7f49f006abe0 0x7f49f006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418516126 0 0) 0x7f49f01406f0 con 0x7f49f01409d0 2023-12-16T21:15:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.103+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49dc003520 con 0x7f49f01409d0 2023-12-16T21:15:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2112287973 0 0) 0x7f49dc003520 con 0x7f49f01409d0 2023-12-16T21:15:28.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 msgr2=0x7f49f01618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 0x7f49f01618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 >> v2:172.21.15.121:3300/0 conn(0x7f49f006abe0 msgr2=0x7f49f006efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 --2- 172.21.15.46:0/1297839387 >> v2:172.21.15.121:3300/0 conn(0x7f49f006abe0 0x7f49f006efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f0142de0 con 0x7f49f01409d0 2023-12-16T21:15:28.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f49e0001670 con 0x7f49f01409d0 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 >> v1:172.21.15.46:6789/0 conn(0x7f49f01409d0 legacy=0x7f49f006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 0x7f49dc003990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49dc003f10 0x7f49dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f0161e20 con 0x7f49dc004d00 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f0143960 con 0x7f49dc003f10 2023-12-16T21:15:28.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49dc00ba00 con 0x7f49f0140310 2023-12-16T21:15:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f68a2640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f60a1640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 0x7f49dc003990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f58a0640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49dc003f10 0x7f49dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f68a2640 1 -- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 msgr2=0x7f49dc003990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f68a2640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f0140310 0x7f49dc003990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f68a2640 1 -- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49dc003f10 msgr2=0x7f49dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f68a2640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49dc003f10 0x7f49dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.104+0000 7f49f68a2640 1 -- 172.21.15.46:0/1297839387 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49dc003520 con 0x7f49dc004d00 2023-12-16T21:15:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.105+0000 7f49f68a2640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49dc005180 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009f90 tx=0x7f49dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a133c8a618bc287 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.105+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e802cc20 con 0x7f49dc004d00 2023-12-16T21:15:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.105+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e800f040 con 0x7f49dc004d00 2023-12-16T21:15:28.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.105+0000 7f49f509f640 1 -- 172.21.15.46:0/1297839387 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e8034440 con 0x7f49dc004d00 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f8305640 1 -- 172.21.15.46:0/1297839387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 msgr2=0x7f49dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f8305640 1 --2- 172.21.15.46:0/1297839387 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49dc005180 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f49e8009f90 tx=0x7f49dc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f68a2640 1 -- 172.21.15.46:0/1297839387 reap_dead start 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f8305640 1 -- 172.21.15.46:0/1297839387 shutdown_connections 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f8305640 1 -- 172.21.15.46:0/1297839387 >> 172.21.15.46:0/1297839387 conn(0x7f49f00b1e50 msgr2=0x7f49f00b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f8305640 1 -- 172.21.15.46:0/1297839387 shutdown_connections 2023-12-16T21:15:28.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.106+0000 7f49f8305640 1 -- 172.21.15.46:0/1297839387 wait complete. 2023-12-16T21:15:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.107+0000 7f49f8305640 1 Processor -- start 2023-12-16T21:15:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.107+0000 7f49f8305640 1 -- start start 2023-12-16T21:15:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f8305640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49f00b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f8305640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f00b18d0 0x7f49f0145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f8305640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f013ba80 0x7f49f013bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f8305640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f01553b0 con 0x7f49dc004d00 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f8305640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f00b0d20 con 0x7f49f00b18d0 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f8305640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49f0154a80 con 0x7f49f013ba80 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49f00b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49f00b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50838/0 (socket says 172.21.15.46:50838) 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 -- 172.21.15.46:0/3945914419 learned_addr learned my addr 172.21.15.46:0/3945914419 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f58a0640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f00b18d0 0x7f49f0145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 -- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f013ba80 msgr2=0x7f49f013bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f013ba80 0x7f49f013bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 -- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f00b18d0 msgr2=0x7f49f0145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f00b18d0 0x7f49f0145db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.108+0000 7f49f60a1640 1 -- 172.21.15.46:0/3945914419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f0154d60 con 0x7f49dc004d00 2023-12-16T21:15:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.109+0000 7f49f60a1640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49f00b0420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0118e0 tx=0x7f49ec011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.109+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec00faf0 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.109+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49ec00fc50 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.109+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49ec014440 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.109+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f0155040 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.109+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f00b0960 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.110+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f0161030 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.110+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f49ec0145a0 con 0x7f49dc004d00 2023-12-16T21:15:28.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.111+0000 7f49e6ffd640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f49d406ae40 0x7f49d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.111+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f49ec089c20 con 0x7f49dc004d00 2023-12-16T21:15:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.111+0000 7f49f58a0640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f49d406ae40 0x7f49d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.112+0000 7f49f58a0640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f49d406ae40 0x7f49d406d300 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f49f006fcc0 tx=0x7f49e0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.115+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f49ec055010 con 0x7f49dc004d00 2023-12-16T21:15:28.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.354+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f49f0069400 con 0x7f49dc004d00 2023-12-16T21:15:28.356 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.354+0000 7f49e6ffd640 1 -- 172.21.15.46:0/3945914419 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5298 (secure 0 0 0) 0x7f49ec022090 con 0x7f49dc004d00 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "auth_dump": [ 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.a", 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lilFjJRAAS3jGgFJPZWkcXnfWFuSkmw==", 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.b", 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5liiXVMhAAJWJpsjXpIJHc/8ySyArSgQ==", 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.c", 2023-12-16T21:15:28.359 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lElZsLRAA5EGpXLzgGQdhS3J+Pf+9gw==", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mds.d", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5l7uY3OhAAwqyF6WkAH8BGy1tWJygLCg==", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.360 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.0", 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC/En5lteX6JxAAOUBR8wDo1UbdDenhHLeSNg==", 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.1", 2023-12-16T21:15:28.361 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDCEn5le0WlJxAAeyeKmj4yjwDCNn0WuoW4Rw==", 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.362 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.2", 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDFEn5l/ZafJhAA72K0MtXsz0kQAa0Hhlrg5g==", 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.3", 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDIEn5l/BX4JRAAcfb0TNdwljS3Hwp6E3EPXw==", 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.363 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.4", 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDOEn5lmZNQLRAAZ4THlfKtWJxeQXvoVShsRg==", 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.364 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.5", 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDREn5lPZlmLBAABoR8+M5fdPUah5hZITIABA==", 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.6", 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDUEn5lkOKRKxAAa5eJiAzXy/L79xL7egwC+g==", 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.365 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "osd.7", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQDXEn5lyQ1RKhAAcFWeiQsS4BaAJ/fIgCiCPA==", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow profile osd", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile osd", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.0", 2023-12-16T21:15:28.366 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w==", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow r", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rw pool=cephfs_data" 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.1", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5l7si2ChAAPixHXNSDvbDoEOOjVYonbQ==", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:15:28.367 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.2", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5l66kFBxAApn9h5R4KlGm9Udlz7FGldw==", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.3", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC+En5l1NSAEBAAqlxrkimPL3Z1z3llw4xr9Q==", 2023-12-16T21:15:28.368 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow r", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow rw", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow rwx" 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "client.admin", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lgH3XBhAA9d1n5heC1V63dyfuAmerGA==", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr": "allow *", 2023-12-16T21:15:28.369 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow *", 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.x", 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lD2F+IRAAD5Xq3cx5Gd9fKtnuYe9HMw==", 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:28.370 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: "entity": "mgr.y", 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: "key": "AQC9En5lgvZiHRAA87oDIk7xwjmYe4gMMg1NTQ==", 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": { 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": "allow *", 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: "mon": "allow profile mgr", 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: "osd": "allow *" 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2023-12-16T21:15:28.371 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f49d406ae40 msgr2=0x7f49d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f49d406ae40 0x7f49d406d300 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f49f006fcc0 tx=0x7f49e0009040 comp rx=0 tx=0).stop 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 msgr2=0x7f49f00b0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49f00b0420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f49ec0118e0 tx=0x7f49ec011da0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 shutdown_connections 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f49d406ae40 0x7f49d406d300 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49f013ba80 0x7f49f013bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49f00b18d0 0x7f49f0145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 --2- 172.21.15.46:0/3945914419 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f49dc004d00 0x7f49f00b0420 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.373 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 >> 172.21.15.46:0/3945914419 conn(0x7f49f00b1e50 msgr2=0x7f49f00b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:28.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 shutdown_connections 2023-12-16T21:15:28.374 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.356+0000 7f49f8305640 1 -- 172.21.15.46:0/3945914419 wait complete. 2023-12-16T21:15:28.374 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.483+0000 7fc02d33e640 1 Processor -- start 2023-12-16T21:15:28.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.483+0000 7fc02d33e640 1 -- start start 2023-12-16T21:15:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.483+0000 7fc02d33e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc0280d2910 0x7fc0280d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.483+0000 7fc02d33e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0280b2b10 con 0x7fc02815d330 2023-12-16T21:15:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.483+0000 7fc02d33e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0280d1590 con 0x7fc0280d3730 2023-12-16T21:15:28.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.483+0000 7fc02d33e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0280d3bc0 con 0x7fc0280d2910 2023-12-16T21:15:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc026ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc02815d330 0x7fc0280b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42248/0 (socket says 172.21.15.46:42248) 2023-12-16T21:15:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc026ffd640 1 -- 172.21.15.46:0/352973493 learned_addr learned my addr 172.21.15.46:0/352973493 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc01ffff640 1 --2- 172.21.15.46:0/352973493 >> v2:172.21.15.121:3300/0 conn(0x7fc0280d2910 0x7fc0280d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc0267fc640 1 -- 172.21.15.46:0/352973493 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741968855 0 0) 0x7fc0280b2b10 con 0x7fc02815d330 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc0267fc640 1 -- 172.21.15.46:0/352973493 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014003520 con 0x7fc02815d330 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc0267fc640 1 -- 172.21.15.46:0/352973493 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729235263 0 0) 0x7fc014003520 con 0x7fc02815d330 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc01ffff640 1 -- 172.21.15.46:0/352973493 >> v1:172.21.15.121:6789/0 conn(0x7fc0280d3730 legacy=0x7fc0280b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc01ffff640 1 -- 172.21.15.46:0/352973493 >> v1:172.21.15.46:6789/0 conn(0x7fc02815d330 legacy=0x7fc0280b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.484+0000 7fc01ffff640 1 -- 172.21.15.46:0/352973493 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280d1870 con 0x7fc0280d2910 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.485+0000 7fc01ffff640 1 --2- 172.21.15.46:0/352973493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280d31f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc0100097a0 tx=0x7fc01002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b6ce49a669426f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.485+0000 7fc0267fc640 1 -- 172.21.15.46:0/352973493 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018001670 con 0x7fc0280d2910 2023-12-16T21:15:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.485+0000 7fc0267fc640 1 -- 172.21.15.46:0/352973493 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01002e050 con 0x7fc0280d2910 2023-12-16T21:15:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.485+0000 7fc0267fc640 1 -- 172.21.15.46:0/352973493 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010030710 con 0x7fc0280d2910 2023-12-16T21:15:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 -- 172.21.15.46:0/352973493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 msgr2=0x7fc0280d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 --2- 172.21.15.46:0/352973493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280d31f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc0100097a0 tx=0x7fc01002b790 comp rx=0 tx=0).stop 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 -- 172.21.15.46:0/352973493 shutdown_connections 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 --2- 172.21.15.46:0/352973493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280d31f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 -- 172.21.15.46:0/352973493 >> 172.21.15.46:0/352973493 conn(0x7fc028072020 msgr2=0x7fc028072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 -- 172.21.15.46:0/352973493 shutdown_connections 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.486+0000 7fc02d33e640 1 -- 172.21.15.46:0/352973493 wait complete. 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.487+0000 7fc02d33e640 1 Processor -- start 2023-12-16T21:15:28.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.487+0000 7fc02d33e640 1 -- start start 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 0x7fc0281f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc02815d330 0x7fc02814c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280c8280 con 0x7fc0280d3730 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0281f3560 con 0x7fc0280d2910 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0280ca780 con 0x7fc02815d330 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 0x7fc0281f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 0x7fc0281f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50840/0 (socket says 172.21.15.46:50840) 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 -- 172.21.15.46:0/896920763 learned_addr learned my addr 172.21.15.46:0/896920763 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc026ffd640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc0277fe640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc02815d330 0x7fc02814c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 -- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc02815d330 msgr2=0x7fc02814c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc02815d330 0x7fc02814c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 -- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 msgr2=0x7fc0280cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 -- 172.21.15.46:0/896920763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010009400 con 0x7fc0280d3730 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01ffff640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 0x7fc0281f3020 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc0100098c0 tx=0x7fc0100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0100044c0 con 0x7fc0280d3730 2023-12-16T21:15:28.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc010030510 con 0x7fc0280d3730 2023-12-16T21:15:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0100308c0 con 0x7fc0280d3730 2023-12-16T21:15:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.488+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0280c84d0 con 0x7fc0280d3730 2023-12-16T21:15:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.489+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0280c8a90 con 0x7fc0280d3730 2023-12-16T21:15:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.490+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc028069400 con 0x7fc0280d3730 2023-12-16T21:15:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.490+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc010030a20 con 0x7fc0280d3730 2023-12-16T21:15:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.490+0000 7fc01f7fe640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc00806ae70 0x7fc00806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.491+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0100aaf10 con 0x7fc0280d3730 2023-12-16T21:15:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.491+0000 7fc026ffd640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc00806ae70 0x7fc00806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.491+0000 7fc026ffd640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc00806ae70 0x7fc00806d330 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc018001eb0 tx=0x7fc018009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.495+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc0100763a0 con 0x7fc0280d3730 2023-12-16T21:15:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.672+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0280ce440 con 0x7fc0280d3730 2023-12-16T21:15:28.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.673+0000 7fc01f7fe640 1 -- 172.21.15.46:0/896920763 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fc010041090 con 0x7fc0280d3730 2023-12-16T21:15:28.672 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:28.673 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":46,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:27.929336+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc00806ae70 msgr2=0x7fc00806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc00806ae70 0x7fc00806d330 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc018001eb0 tx=0x7fc018009040 comp rx=0 tx=0).stop 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 msgr2=0x7fc0281f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 0x7fc0281f3020 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fc0100098c0 tx=0x7fc0100046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 shutdown_connections 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc00806ae70 0x7fc00806d330 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0280d3730 0x7fc0281f3020 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc02815d330 0x7fc02814c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 --2- 172.21.15.46:0/896920763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0280d2910 0x7fc0280cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 >> 172.21.15.46:0/896920763 conn(0x7fc028072020 msgr2=0x7fc028072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:28.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 shutdown_connections 2023-12-16T21:15:28.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.675+0000 7fc02d33e640 1 -- 172.21.15.46:0/896920763 wait complete. 2023-12-16T21:15:28.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:28.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:28.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.796+0000 7f7c4facf640 1 Processor -- start 2023-12-16T21:15:28.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.796+0000 7f7c4facf640 1 -- start start 2023-12-16T21:15:28.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4facf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c4806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4facf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c480ad640 0x7f7c4806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4facf640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c480adae0 con 0x7f7c480aeb80 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4facf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c480addc0 con 0x7f7c480ad640 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4facf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480ae0a0 con 0x7f7c4815d1d0 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c480ad640 0x7f7c4806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7c480ad640 0x7f7c4806ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50846/0 (socket says 172.21.15.46:50846) 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 -- 172.21.15.46:0/4258480570 learned_addr learned my addr 172.21.15.46:0/4258480570 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 -- 172.21.15.46:0/4258480570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 msgr2=0x7f7c4806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 --2- 172.21.15.46:0/4258480570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c4806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 -- 172.21.15.46:0/4258480570 >> v1:172.21.15.121:6789/0 conn(0x7f7c480aeb80 legacy=0x7f7c48160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:28.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.797+0000 7f7c4d06a640 1 -- 172.21.15.46:0/4258480570 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480ae380 con 0x7f7c480ad640 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.798+0000 7f7c4d06a640 1 --2- 172.21.15.46:0/4258480570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c4806ba20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009870 tx=0x7f7c3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a93b4748410efd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.798+0000 7f7c4c869640 1 -- 172.21.15.46:0/4258480570 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38031640 con 0x7f7c480ad640 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.798+0000 7f7c4c869640 1 -- 172.21.15.46:0/4258480570 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c380317a0 con 0x7f7c480ad640 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.798+0000 7f7c4c869640 1 -- 172.21.15.46:0/4258480570 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38031b30 con 0x7f7c480ad640 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 -- 172.21.15.46:0/4258480570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 msgr2=0x7f7c4806ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 --2- 172.21.15.46:0/4258480570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c4806ba20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009870 tx=0x7f7c3802b8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 -- 172.21.15.46:0/4258480570 shutdown_connections 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 --2- 172.21.15.46:0/4258480570 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c4806ba20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 --2- 172.21.15.46:0/4258480570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c4806b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 -- 172.21.15.46:0/4258480570 >> 172.21.15.46:0/4258480570 conn(0x7f7c48070e70 msgr2=0x7f7c48071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 -- 172.21.15.46:0/4258480570 shutdown_connections 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.799+0000 7f7c4facf640 1 -- 172.21.15.46:0/4258480570 wait complete. 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.800+0000 7f7c4facf640 1 Processor -- start 2023-12-16T21:15:28.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.800+0000 7f7c4facf640 1 -- start start 2023-12-16T21:15:28.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4facf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c48160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4facf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c480aeb80 0x7f7c48152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4facf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c481528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4facf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cd700 con 0x7f7c480ad640 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4facf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480ccdd0 con 0x7f7c480aeb80 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4facf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cd0b0 con 0x7f7c4815d1d0 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c48160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c48160280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50848/0 (socket says 172.21.15.46:50848) 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 -- 172.21.15.46:0/3960991083 learned_addr learned my addr 172.21.15.46:0/3960991083 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d06a640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c480aeb80 0x7f7c48152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4e06c640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c481528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 -- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 msgr2=0x7f7c481528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c481528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 -- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c480aeb80 msgr2=0x7f7c48152360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c480aeb80 0x7f7c48152360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.801+0000 7f7c4d86b640 1 -- 172.21.15.46:0/3960991083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c380094d0 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.802+0000 7f7c4d86b640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c48160280 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000b410 tx=0x7f7c4000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.802+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40011830 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.802+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c400119b0 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.802+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c480cd390 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.802+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40011d60 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.802+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4815e590 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.803+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c48069400 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.803+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7c4001a400 con 0x7f7c480ad640 2023-12-16T21:15:28.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.804+0000 7f7c3e7fc640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7c2c06ae40 0x7f7c2c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:28.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.804+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c40089230 con 0x7f7c480ad640 2023-12-16T21:15:28.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.804+0000 7f7c4d06a640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7c2c06ae40 0x7f7c2c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:28.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.804+0000 7f7c4d06a640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7c2c06ae40 0x7f7c2c06d300 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c38036000 tx=0x7f7c3803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:28.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.808+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7c40054770 con 0x7f7c480ad640 2023-12-16T21:15:28.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.986+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7c4815b9b0 con 0x7f7c480ad640 2023-12-16T21:15:28.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.987+0000 7f7c3e7fc640 1 -- 172.21.15.46:0/3960991083 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c40057d90 con 0x7f7c480ad640 2023-12-16T21:15:28.987 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7c2c06ae40 msgr2=0x7f7c2c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7c2c06ae40 0x7f7c2c06d300 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c38036000 tx=0x7f7c3803c040 comp rx=0 tx=0).stop 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 msgr2=0x7f7c48160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c48160280 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000b410 tx=0x7f7c4000b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 shutdown_connections 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7c2c06ae40 0x7f7c2c06d300 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c4815d1d0 0x7f7c481528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c480aeb80 0x7f7c48152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 --2- 172.21.15.46:0/3960991083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7c480ad640 0x7f7c48160280 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:28.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 >> 172.21.15.46:0/3960991083 conn(0x7f7c48070e70 msgr2=0x7f7c4815bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:28.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 shutdown_connections 2023-12-16T21:15:28.990 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:28.989+0000 7f7c4facf640 1 -- 172.21.15.46:0/3960991083 wait complete. 2023-12-16T21:15:28.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-16T21:15:29.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.105+0000 7fe814044640 1 Processor -- start 2023-12-16T21:15:29.105 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.105+0000 7fe814044640 1 -- start start 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.106+0000 7fe814044640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe80c15d2d0 0x7fe80c0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.106+0000 7fe814044640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c0d1bb0 con 0x7fe80c0d2040 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.106+0000 7fe814044640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe80c15d6b0 con 0x7fe80c0ce150 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.106+0000 7fe814044640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe80c0ce700 con 0x7fe80c15d2d0 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.106+0000 7fe8115df640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe80c0d2040 0x7fe80c0cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:42258/0 (socket says 172.21.15.46:42258) 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.106+0000 7fe8115df640 1 -- 172.21.15.46:0/2674316207 learned_addr learned my addr 172.21.15.46:0/2674316207 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686152584 0 0) 0x7fe80c0d1bb0 con 0x7fe80c0d2040 2023-12-16T21:15:29.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f4003520 con 0x7fe80c0d2040 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe811de0640 1 --2- 172.21.15.46:0/2674316207 >> v2:172.21.15.121:3300/0 conn(0x7fe80c15d2d0 0x7fe80c0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648244727 0 0) 0x7fe7f4003520 con 0x7fe80c0d2040 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 >> v2:172.21.15.121:3300/0 conn(0x7fe80c15d2d0 msgr2=0x7fe80c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 --2- 172.21.15.46:0/2674316207 >> v2:172.21.15.121:3300/0 conn(0x7fe80c15d2d0 0x7fe80c0b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 >> v1:172.21.15.121:6789/0 conn(0x7fe80c0ce150 legacy=0x7fe80c0b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0d1280 con 0x7fe80c0d2040 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fe800001670 con 0x7fe80c0d2040 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 >> v1:172.21.15.46:6789/0 conn(0x7fe80c0d2040 legacy=0x7fe80c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7f4003970 0x7fe7f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7f40047c0 0x7fe7f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe7f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0ce700 con 0x7fe7f4005180 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f400c020 con 0x7fe7f40047c0 2023-12-16T21:15:29.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f400c1e0 con 0x7fe7f4003970 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe8125e1640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe7f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe8115df640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7f40047c0 0x7fe7f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe811de0640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7f4003970 0x7fe7f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe8125e1640 1 -- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7f4003970 msgr2=0x7fe7f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe8125e1640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7f4003970 0x7fe7f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.107+0000 7fe8125e1640 1 -- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7f40047c0 msgr2=0x7fe7f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe8125e1640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7f40047c0 0x7fe7f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe8125e1640 1 -- 172.21.15.46:0/2674316207 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4003520 con 0x7fe7f4005180 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe8125e1640 1 -- 172.21.15.46:0/2674316207 reap_dead start 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe8125e1640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe7f400b980 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc001e50 tx=0x7fe7f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d00c2b745dd958a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc003b70 con 0x7fe7f4005180 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc003d10 con 0x7fe7f4005180 2023-12-16T21:15:29.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.108+0000 7fe810dde640 1 -- 172.21.15.46:0/2674316207 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc005310 con 0x7fe7f4005180 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 -- 172.21.15.46:0/2674316207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 msgr2=0x7fe7f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe7f400b980 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc001e50 tx=0x7fe7f40057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 -- 172.21.15.46:0/2674316207 shutdown_connections 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 --2- 172.21.15.46:0/2674316207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe7f400b980 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 -- 172.21.15.46:0/2674316207 >> 172.21.15.46:0/2674316207 conn(0x7fe80c072050 msgr2=0x7fe80c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 -- 172.21.15.46:0/2674316207 shutdown_connections 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.109+0000 7fe814044640 1 -- 172.21.15.46:0/2674316207 wait complete. 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.110+0000 7fe814044640 1 Processor -- start 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.110+0000 7fe814044640 1 -- start start 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe814044640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe80c1f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe814044640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe80c14e500 0x7fe80c15e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe814044640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe80c14e940 0x7fe80c15e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe814044640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0ccdb0 con 0x7fe7f4005180 2023-12-16T21:15:29.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe814044640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0ca450 con 0x7fe80c14e940 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe814044640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c0ca730 con 0x7fe80c14e500 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe80c1f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe80c1f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50864/0 (socket says 172.21.15.46:50864) 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 -- 172.21.15.46:0/1520423246 learned_addr learned my addr 172.21.15.46:0/1520423246 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe8115df640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe80c14e500 0x7fe80c15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe8125e1640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe80c14e940 0x7fe80c15e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 -- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe80c14e500 msgr2=0x7fe80c15e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe80c14e500 0x7fe80c15e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 -- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe80c14e940 msgr2=0x7fe80c15e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe80c14e940 0x7fe80c15e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe811de0640 1 -- 172.21.15.46:0/1520423246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c0caa10 con 0x7fe7f4005180 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe8115df640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe80c14e500 0x7fe80c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.111+0000 7fe8125e1640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe80c14e940 0x7fe80c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:29.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.112+0000 7fe811de0640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe80c1f3d10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe8040118e0 tx=0x7fe804011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.112+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80400faf0 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.112+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80400fc50 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.112+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c0cacf0 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.112+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe804014440 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.112+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c15f520 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.113+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80c0ca1d0 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.113+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe80401d400 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.114+0000 7fe80a7fc640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe7f806ae40 0x7fe7f806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.114+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe804089ba0 con 0x7fe7f4005180 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.114+0000 7fe8115df640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe7f806ae40 0x7fe7f806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.115+0000 7fe8115df640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe7f806ae40 0x7fe7f806d300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe80c15ef30 tx=0x7fe800009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:29.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.118+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe804055030 con 0x7fe7f4005180 2023-12-16T21:15:29.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.298+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe80c069400 con 0x7fe7f4005180 2023-12-16T21:15:29.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.299+0000 7fe80a7fc640 1 -- 172.21.15.46:0/1520423246 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v46) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe804058650 con 0x7fe7f4005180 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout: { 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:29.299 INFO:teuthology.orchestra.run.smithi046.stdout:] 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe7f806ae40 msgr2=0x7fe7f806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe7f806ae40 0x7fe7f806d300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe80c15ef30 tx=0x7fe800009040 comp rx=0 tx=0).stop 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 msgr2=0x7fe80c1f3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe80c1f3d10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe8040118e0 tx=0x7fe804011da0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 shutdown_connections 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe7f806ae40 0x7fe7f806d300 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe80c14e940 0x7fe80c15e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe80c14e500 0x7fe80c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 --2- 172.21.15.46:0/1520423246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe7f4005180 0x7fe80c1f3d10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 >> 172.21.15.46:0/1520423246 conn(0x7fe80c072050 msgr2=0x7fe80c0b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 shutdown_connections 2023-12-16T21:15:29.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.301+0000 7fe814044640 1 -- 172.21.15.46:0/1520423246 wait complete. 2023-12-16T21:15:29.311 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-16T21:15:29.311 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-16T21:15:29.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.422+0000 7fd52611d640 1 Processor -- start 2023-12-16T21:15:29.422 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.423+0000 7fd52611d640 1 -- start start 2023-12-16T21:15:29.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.423+0000 7fd52611d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd520140310 0x7fd5201618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.423+0000 7fd52611d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd52006abe0 0x7fd52006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.423+0000 7fd52611d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5201406f0 con 0x7fd5201409d0 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.423+0000 7fd52611d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd520161e20 con 0x7fd52006abe0 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.423+0000 7fd52611d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd520143960 con 0x7fd520140310 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd52006abe0 0x7fd52006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd52006abe0 0x7fd52006efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50874/0 (socket says 172.21.15.46:50874) 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 -- 172.21.15.46:0/1271511768 learned_addr learned my addr 172.21.15.46:0/1271511768 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51f7fe640 1 --2- 172.21.15.46:0/1271511768 >> v2:172.21.15.121:3300/0 conn(0x7fd520140310 0x7fd5201618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 -- 172.21.15.46:0/1271511768 >> v2:172.21.15.121:3300/0 conn(0x7fd520140310 msgr2=0x7fd5201618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 --2- 172.21.15.46:0/1271511768 >> v2:172.21.15.121:3300/0 conn(0x7fd520140310 0x7fd5201618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 -- 172.21.15.46:0/1271511768 >> v1:172.21.15.121:6789/0 conn(0x7fd5201409d0 legacy=0x7fd52006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:29.424 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.424+0000 7fd51ffff640 1 -- 172.21.15.46:0/1271511768 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520142de0 con 0x7fd52006abe0 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd51ffff640 1 --2- 172.21.15.46:0/1271511768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd52006abe0 0x7fd52006efb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd5200b5840 tx=0x7fd51802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1912c9485c6ea3d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd51e7fc640 1 -- 172.21.15.46:0/1271511768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51802c9f0 con 0x7fd52006abe0 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd51e7fc640 1 -- 172.21.15.46:0/1271511768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51802cb50 con 0x7fd52006abe0 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd51e7fc640 1 -- 172.21.15.46:0/1271511768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5180343f0 con 0x7fd52006abe0 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 -- 172.21.15.46:0/1271511768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd52006abe0 msgr2=0x7fd52006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 --2- 172.21.15.46:0/1271511768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd52006abe0 0x7fd52006efb0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd5200b5840 tx=0x7fd51802be50 comp rx=0 tx=0).stop 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 -- 172.21.15.46:0/1271511768 shutdown_connections 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 --2- 172.21.15.46:0/1271511768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd52006abe0 0x7fd52006efb0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 --2- 172.21.15.46:0/1271511768 >> v2:172.21.15.121:3300/0 conn(0x7fd520140310 0x7fd5201618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 -- 172.21.15.46:0/1271511768 >> 172.21.15.46:0/1271511768 conn(0x7fd5200b1e50 msgr2=0x7fd5200b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 -- 172.21.15.46:0/1271511768 shutdown_connections 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.425+0000 7fd52611d640 1 -- 172.21.15.46:0/1271511768 wait complete. 2023-12-16T21:15:29.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.426+0000 7fd52611d640 1 Processor -- start 2023-12-16T21:15:29.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 -- start start 2023-12-16T21:15:29.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd52006abe0 0x7fd52006d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd520140310 0x7fd520131770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 0x7fd520131cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52012f600 con 0x7fd5201409d0 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201321f0 con 0x7fd520140310 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd52611d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5201324a0 con 0x7fd52006abe0 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 0x7fd520131cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd51ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 0x7fd520131cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50878/0 (socket says 172.21.15.46:50878) 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.427+0000 7fd51ffff640 1 -- 172.21.15.46:0/1938189128 learned_addr learned my addr 172.21.15.46:0/1938189128 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51effd640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd520140310 0x7fd520131770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51f7fe640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd52006abe0 0x7fd52006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51ffff640 1 -- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd52006abe0 msgr2=0x7fd52006d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51ffff640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd52006abe0 0x7fd52006d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51ffff640 1 -- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd520140310 msgr2=0x7fd520131770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51ffff640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd520140310 0x7fd520131770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51ffff640 1 -- 172.21.15.46:0/1938189128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd518009b30 con 0x7fd5201409d0 2023-12-16T21:15:29.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd51ffff640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 0x7fd520131cb0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd518036040 tx=0x7fd518004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:29.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51803e450 con 0x7fd5201409d0 2023-12-16T21:15:29.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51803e5d0 con 0x7fd5201409d0 2023-12-16T21:15:29.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd52012f8a0 con 0x7fd5201409d0 2023-12-16T21:15:29.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51803e980 con 0x7fd5201409d0 2023-12-16T21:15:29.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.428+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd52012fe60 con 0x7fd5201409d0 2023-12-16T21:15:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.429+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd520130050 con 0x7fd5201409d0 2023-12-16T21:15:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.430+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd51803eae0 con 0x7fd5201409d0 2023-12-16T21:15:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.430+0000 7fd507fff640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd50c06ae40 0x7fd50c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.430+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd5180ac1b0 con 0x7fd5201409d0 2023-12-16T21:15:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.430+0000 7fd51f7fe640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd50c06ae40 0x7fd50c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:29.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.431+0000 7fd51f7fe640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd50c06ae40 0x7fd50c06d300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd510006fd0 tx=0x7fd510008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:29.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.434+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd518077660 con 0x7fd5201409d0 2023-12-16T21:15:29.610 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:29.610+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd520069400 con 0x7fd5201409d0 2023-12-16T21:15:30.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.616+0000 7fd507fff640 1 -- 172.21.15.46:0/1938189128 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd51807ac80 con 0x7fd5201409d0 2023-12-16T21:15:30.616 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2023-12-16T21:15:30.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.620+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd50c06ae40 msgr2=0x7fd50c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.620+0000 7fd52611d640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd50c06ae40 0x7fd50c06d300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd510006fd0 tx=0x7fd510008040 comp rx=0 tx=0).stop 2023-12-16T21:15:30.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.620+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 msgr2=0x7fd520131cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.620+0000 7fd52611d640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 0x7fd520131cb0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd518036040 tx=0x7fd518004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 shutdown_connections 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd50c06ae40 0x7fd50c06d300 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5201409d0 0x7fd520131cb0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd520140310 0x7fd520131770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 --2- 172.21.15.46:0/1938189128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd52006abe0 0x7fd52006d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 >> 172.21.15.46:0/1938189128 conn(0x7fd5200b1e50 msgr2=0x7fd52015eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 shutdown_connections 2023-12-16T21:15:30.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.621+0000 7fd52611d640 1 -- 172.21.15.46:0/1938189128 wait complete. 2023-12-16T21:15:30.637 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-16T21:15:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.757+0000 7f0ac5505640 1 Processor -- start 2023-12-16T21:15:30.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.757+0000 7f0ac5505640 1 -- start start 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0ac5505640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 0x7f0ac00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0ac5505640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00cf660 con 0x7f0ac00b4060 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0ac5505640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac0161170 con 0x7f0ac0146010 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0ac5505640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0161450 con 0x7f0ac0146d70 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abf7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f0ac00b4060 0x7f0ac006c6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34412/0 (socket says 172.21.15.46:34412) 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abf7fe640 1 -- 172.21.15.46:0/3274736414 learned_addr learned my addr 172.21.15.46:0/3274736414 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 0x7f0ac00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484239014 0 0) 0x7f0ac00cf660 con 0x7f0ac00b4060 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0aac003520 con 0x7f0ac00b4060 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426200347 0 0) 0x7f0ac0161170 con 0x7f0ac0146010 2023-12-16T21:15:30.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ac00cf660 con 0x7f0ac0146010 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952413495 0 0) 0x7f0aac003520 con 0x7f0ac00b4060 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 msgr2=0x7f0ac00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 0x7f0ac00b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 >> v1:172.21.15.121:6789/0 conn(0x7f0ac0146010 legacy=0x7f0ac00cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0161730 con 0x7f0ac00b4060 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.758+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f0ab80019f0 con 0x7f0ac00b4060 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 >> v1:172.21.15.46:6789/0 conn(0x7f0ac00b4060 legacy=0x7f0ac006c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aac003b30 0x7f0aac003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0aac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 0x7f0aac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0161450 con 0x7f0aac0047c0 2023-12-16T21:15:30.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac00ba30 con 0x7f0aac003b30 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac00bbf0 con 0x7f0ac0146d70 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0aac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abeffd640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aac003b30 0x7f0aac003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0abf7fe640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 0x7f0aac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 msgr2=0x7f0aac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac0146d70 0x7f0aac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aac003b30 msgr2=0x7f0aac003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aac003b30 0x7f0aac003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.759+0000 7f0ab7fff640 1 -- 172.21.15.46:0/3274736414 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac003520 con 0x7f0aac0047c0 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0aac004c40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0147150 tx=0x7f0aac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a08a9478d20b4632 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa802c630 con 0x7f0aac0047c0 2023-12-16T21:15:30.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa802c790 con 0x7f0aac0047c0 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0abe7fc640 1 -- 172.21.15.46:0/3274736414 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa802cb20 con 0x7f0aac0047c0 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0ac5505640 1 -- 172.21.15.46:0/3274736414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 msgr2=0x7f0aac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0ac5505640 1 --2- 172.21.15.46:0/3274736414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0aac004c40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0147150 tx=0x7f0aac0057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.760+0000 7f0abf7fe640 1 -- 172.21.15.46:0/3274736414 reap_dead start 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.761+0000 7f0ac5505640 1 -- 172.21.15.46:0/3274736414 shutdown_connections 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.761+0000 7f0ac5505640 1 -- 172.21.15.46:0/3274736414 >> 172.21.15.46:0/3274736414 conn(0x7f0ac006a4d0 msgr2=0x7f0ac006a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.761+0000 7f0ac5505640 1 -- 172.21.15.46:0/3274736414 shutdown_connections 2023-12-16T21:15:30.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.761+0000 7f0ac5505640 1 -- 172.21.15.46:0/3274736414 wait complete. 2023-12-16T21:15:30.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.762+0000 7f0ac5505640 1 Processor -- start 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.762+0000 7f0ac5505640 1 -- start start 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.762+0000 7f0ac5505640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0ac00ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.762+0000 7f0ac5505640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac01441c0 0x7f0ac00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.762+0000 7f0ac5505640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac00ca3e0 0x7f0ac015c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ac5505640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0144f70 con 0x7f0aac0047c0 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ac5505640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01446d0 con 0x7f0ac01441c0 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ac5505640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01449b0 con 0x7f0ac00ca3e0 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0ac00ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0ac00ce360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59626/0 (socket says 172.21.15.46:59626) 2023-12-16T21:15:30.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 -- 172.21.15.46:0/1916885083 learned_addr learned my addr 172.21.15.46:0/1916885083 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac01441c0 0x7f0ac00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abf7fe640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac00ca3e0 0x7f0ac015c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 -- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac00ca3e0 msgr2=0x7f0ac015c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac00ca3e0 0x7f0ac015c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 -- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac01441c0 msgr2=0x7f0ac00b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac01441c0 0x7f0ac00b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 -- 172.21.15.46:0/1916885083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0146570 con 0x7f0aac0047c0 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abf7fe640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac00ca3e0 0x7f0ac015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:30.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0abeffd640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0ac00ce360 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000f7a0 tx=0x7f0ab000fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000fe50 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab00052e0 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000d970 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac0144c90 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.763+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00ce8a0 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.764+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0142ff0 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.765+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f0ab000dad0 con 0x7f0aac0047c0 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.765+0000 7f0ab77fe640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa006ae40 0x7f0aa006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:30.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.765+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f0ab0089450 con 0x7f0aac0047c0 2023-12-16T21:15:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.765+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa006ae40 0x7f0aa006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.766+0000 7f0ab7fff640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa006ae40 0x7f0aa006d300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0147150 tx=0x7f0aa8030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:30.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.770+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f0ac0142ff0 con 0x7f0aac0047c0 2023-12-16T21:15:30.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:30.945+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0ac0142ff0 con 0x7f0aac0047c0 2023-12-16T21:15:31.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.619+0000 7f0ab77fe640 1 -- 172.21.15.46:0/1916885083 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v48) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0ac0142ff0 con 0x7f0aac0047c0 2023-12-16T21:15:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.623+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa006ae40 msgr2=0x7f0aa006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.623+0000 7f0ac5505640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa006ae40 0x7f0aa006d300 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0147150 tx=0x7f0aa8030000 comp rx=0 tx=0).stop 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.623+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 msgr2=0x7f0ac00ce360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.623+0000 7f0ac5505640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0ac00ce360 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0ab000f7a0 tx=0x7f0ab000fc60 comp rx=0 tx=0).stop 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.624+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 shutdown_connections 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.624+0000 7f0ac5505640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f0aa006ae40 0x7f0aa006d300 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.624+0000 7f0ac5505640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac00ca3e0 0x7f0ac015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.624+0000 7f0ac5505640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac01441c0 0x7f0ac00b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.624+0000 7f0ac5505640 1 --2- 172.21.15.46:0/1916885083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aac0047c0 0x7f0ac00ce360 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.624+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 >> 172.21.15.46:0/1916885083 conn(0x7f0ac006a4d0 msgr2=0x7f0ac01603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.625+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 shutdown_connections 2023-12-16T21:15:31.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.628+0000 7f0ac5505640 1 -- 172.21.15.46:0/1916885083 wait complete. 2023-12-16T21:15:31.651 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-16T21:15:31.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.764+0000 7f84817c9640 1 Processor -- start 2023-12-16T21:15:31.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f84817c9640 1 -- start start 2023-12-16T21:15:31.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f84817c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f847c15d2d0 0x7f847c0b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f84817c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f847c160f70 0x7f847c0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f84817c9640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847c1614e0 con 0x7f847c0ce480 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f84817c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c1617c0 con 0x7f847c15d2d0 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f84817c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847c161aa0 con 0x7f847c160f70 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f847affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f847c15d2d0 0x7f847c0b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f847affd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f847c15d2d0 0x7f847c0b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59630/0 (socket says 172.21.15.46:59630) 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.765+0000 7f847affd640 1 -- 172.21.15.46:0/986478208 learned_addr learned my addr 172.21.15.46:0/986478208 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.766+0000 7f847a7fc640 1 --2- 172.21.15.46:0/986478208 >> v2:172.21.15.121:3300/0 conn(0x7f847c160f70 0x7f847c0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.766+0000 7f847affd640 1 -- 172.21.15.46:0/986478208 >> v2:172.21.15.121:3300/0 conn(0x7f847c160f70 msgr2=0x7f847c0cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.766+0000 7f847affd640 1 --2- 172.21.15.46:0/986478208 >> v2:172.21.15.121:3300/0 conn(0x7f847c160f70 0x7f847c0cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.766+0000 7f847affd640 1 -- 172.21.15.46:0/986478208 >> v1:172.21.15.46:6789/0 conn(0x7f847c0ce480 legacy=0x7f847c06b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.766+0000 7f847affd640 1 -- 172.21.15.46:0/986478208 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c161d80 con 0x7f847c15d2d0 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.766+0000 7f847a7fc640 1 --2- 172.21.15.46:0/986478208 >> v2:172.21.15.121:3300/0 conn(0x7f847c160f70 0x7f847c0cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f847affd640 1 --2- 172.21.15.46:0/986478208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c15d2d0 0x7f847c0b8b90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f847c0cdb80 tx=0x7f847402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f941db52c1b8af server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:31.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f8479ffb640 1 -- 172.21.15.46:0/986478208 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474030860 con 0x7f847c15d2d0 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f8479ffb640 1 -- 172.21.15.46:0/986478208 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84740309c0 con 0x7f847c15d2d0 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f8479ffb640 1 -- 172.21.15.46:0/986478208 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474030d50 con 0x7f847c15d2d0 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 -- 172.21.15.46:0/986478208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c15d2d0 msgr2=0x7f847c0b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 --2- 172.21.15.46:0/986478208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c15d2d0 0x7f847c0b8b90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f847c0cdb80 tx=0x7f847402b9a0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 -- 172.21.15.46:0/986478208 shutdown_connections 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 --2- 172.21.15.46:0/986478208 >> v2:172.21.15.121:3300/0 conn(0x7f847c160f70 0x7f847c0cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 --2- 172.21.15.46:0/986478208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c15d2d0 0x7f847c0b8b90 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 -- 172.21.15.46:0/986478208 >> 172.21.15.46:0/986478208 conn(0x7f847c072230 msgr2=0x7f847c0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 -- 172.21.15.46:0/986478208 shutdown_connections 2023-12-16T21:15:31.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.767+0000 7f84817c9640 1 -- 172.21.15.46:0/986478208 wait complete. 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 Processor -- start 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 -- start start 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 0x7f847c158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f847c15d2d0 0x7f847c0cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c160f70 0x7f847c15b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c158c80 con 0x7f847c0ce480 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0ca470 con 0x7f847c160f70 2023-12-16T21:15:31.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.769+0000 7f84817c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847c0ca750 con 0x7f847c15d2d0 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 0x7f847c158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 0x7f847c158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59636/0 (socket says 172.21.15.46:59636) 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 -- 172.21.15.46:0/1370450588 learned_addr learned my addr 172.21.15.46:0/1370450588 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847b7fe640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c160f70 0x7f847c15b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847a7fc640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f847c15d2d0 0x7f847c0cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 -- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f847c15d2d0 msgr2=0x7f847c0cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f847c15d2d0 0x7f847c0cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 -- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c160f70 msgr2=0x7f847c15b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c160f70 0x7f847c15b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 -- 172.21.15.46:0/1370450588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84740094d0 con 0x7f847c0ce480 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f847affd640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 0x7f847c158740 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f847402b9e0 tx=0x7f8474004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474039d60 con 0x7f847c0ce480 2023-12-16T21:15:31.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8474043400 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847c0caa30 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84740437b0 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.770+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847c0caff0 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.771+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f847c069400 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.772+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8474043910 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.772+0000 7f84737fe640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f845c06ae40 0x7f845c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.772+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f847403e070 con 0x7f847c0ce480 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.773+0000 7f847a7fc640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f845c06ae40 0x7f845c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:31.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.773+0000 7f847a7fc640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f845c06ae40 0x7f845c06d300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c6b10 tx=0x7f8464008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:31.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.777+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8474077a60 con 0x7f847c0ce480 2023-12-16T21:15:31.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:31.992+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f847c0d0aa0 con 0x7f847c0ce480 2023-12-16T21:15:32.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.626+0000 7f84737fe640 1 -- 172.21.15.46:0/1370450588 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v18) v1 ==== 172+0+0 (secure 0 0 0) 0x7f847407b080 con 0x7f847c0ce480 2023-12-16T21:15:32.626 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 7 and data pool 8 2023-12-16T21:15:32.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.632+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f845c06ae40 msgr2=0x7f845c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.632+0000 7f84817c9640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f845c06ae40 0x7f845c06d300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c6b10 tx=0x7f8464008040 comp rx=0 tx=0).stop 2023-12-16T21:15:32.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.632+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 msgr2=0x7f847c158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.632 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.632+0000 7f84817c9640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 0x7f847c158740 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f847402b9e0 tx=0x7f8474004730 comp rx=0 tx=0).stop 2023-12-16T21:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.634+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 shutdown_connections 2023-12-16T21:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.634+0000 7f84817c9640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f845c06ae40 0x7f845c06d300 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.634+0000 7f84817c9640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f847c160f70 0x7f847c15b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.634+0000 7f84817c9640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f847c15d2d0 0x7f847c0cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.634+0000 7f84817c9640 1 --2- 172.21.15.46:0/1370450588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f847c0ce480 0x7f847c158740 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.634 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.634+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 >> 172.21.15.46:0/1370450588 conn(0x7f847c072230 msgr2=0x7f847c0d19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:32.635 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.636+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 shutdown_connections 2023-12-16T21:15:32.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.637+0000 7f84817c9640 1 -- 172.21.15.46:0/1370450588 wait complete. 2023-12-16T21:15:32.666 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:32.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.778+0000 7f961a548640 1 Processor -- start 2023-12-16T21:15:32.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.778+0000 7f961a548640 1 -- start start 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f961a548640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 0x7f96140cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f961a548640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9614071770 con 0x7f9614144980 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f961a548640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96140ccdc0 con 0x7f96140b4370 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f961a548640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961415ced0 con 0x7f96140ceb70 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9613fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f9614144980 0x7f961415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34426/0 (socket says 172.21.15.46:34426) 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9613fff640 1 -- 172.21.15.46:0/1412099264 learned_addr learned my addr 172.21.15.46:0/1412099264 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191280353 0 0) 0x7f9614071770 con 0x7f9614144980 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f96137fe640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 0x7f96140cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9600003520 con 0x7f9614144980 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4034611530 0 0) 0x7f9600003520 con 0x7f9614144980 2023-12-16T21:15:32.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 msgr2=0x7f96140cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 0x7f96140cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 >> v1:172.21.15.121:6789/0 conn(0x7f96140b4370 legacy=0x7f9614071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140cd0a0 con 0x7f9614144980 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f95fc001670 con 0x7f9614144980 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.779+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 >> v1:172.21.15.46:6789/0 conn(0x7f9614144980 legacy=0x7f961415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9612ffd640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 0x7f96000039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9612ffd640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9600003f20 0x7f96000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9612ffd640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9600004d00 0x7f9600005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961415ced0 con 0x7f9600003f20 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960000ba60 con 0x7f9600004d00 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960000bc20 con 0x7f96140ceb70 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f96137fe640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9600003f20 0x7f96000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9613fff640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 0x7f96000039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f9618ae5640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9600004d00 0x7f9600005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f96137fe640 1 -- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 msgr2=0x7f96000039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f96137fe640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96140ceb70 0x7f96000039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f96137fe640 1 -- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9600004d00 msgr2=0x7f9600005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f96137fe640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9600004d00 0x7f9600005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.780+0000 7f96137fe640 1 -- 172.21.15.46:0/1412099264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9600003520 con 0x7f9600003f20 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f96137fe640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9600003f20 0x7f96000047c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9604009a90 tx=0x7f96000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7017dbde9cd971 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960402c680 con 0x7f9600003f20 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960402c7e0 con 0x7f9600003f20 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f9612ffd640 1 -- 172.21.15.46:0/1412099264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960402cb70 con 0x7f9600003f20 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f961a548640 1 -- 172.21.15.46:0/1412099264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9600003f20 msgr2=0x7f96000047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f961a548640 1 --2- 172.21.15.46:0/1412099264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9600003f20 0x7f96000047c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9604009a90 tx=0x7f96000057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f9618ae5640 1 -- 172.21.15.46:0/1412099264 reap_dead start 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f961a548640 1 -- 172.21.15.46:0/1412099264 shutdown_connections 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f961a548640 1 -- 172.21.15.46:0/1412099264 >> 172.21.15.46:0/1412099264 conn(0x7f961406a4d0 msgr2=0x7f961406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:32.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f961a548640 1 -- 172.21.15.46:0/1412099264 shutdown_connections 2023-12-16T21:15:32.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.781+0000 7f961a548640 1 -- 172.21.15.46:0/1412099264 wait complete. 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 Processor -- start 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 -- start start 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9600003f20 0x7f96140b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96141548c0 0x7f96140b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 0x7f961415c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140b2d50 con 0x7f96140b28b0 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140b3030 con 0x7f96141548c0 2023-12-16T21:15:32.787 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.783+0000 7f961a548640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96140b3310 con 0x7f9600003f20 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 0x7f961415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 0x7f961415c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59652/0 (socket says 172.21.15.46:59652) 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 -- 172.21.15.46:0/122088816 learned_addr learned my addr 172.21.15.46:0/122088816 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f96137fe640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96141548c0 0x7f96140b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9613fff640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9600003f20 0x7f96140b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 -- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9600003f20 msgr2=0x7f96140b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9600003f20 0x7f96140b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 -- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96141548c0 msgr2=0x7f96140b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96141548c0 0x7f96140b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 -- 172.21.15.46:0/122088816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96140b35f0 con 0x7f96140b28b0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9613fff640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9600003f20 0x7f96140b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9618ae5640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 0x7f961415c240 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f961406e5c0 tx=0x7f960c00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c006d60 con 0x7f96140b28b0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c003970 con 0x7f96140b28b0 2023-12-16T21:15:32.788 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96140b38d0 con 0x7f96140b28b0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c003d20 con 0x7f96140b28b0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.784+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96140a8c20 con 0x7f96140b28b0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.785+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614066f90 con 0x7f96140b28b0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.786+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f960c005070 con 0x7f96140b28b0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.786+0000 7f9610ff9640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f95f406ae40 0x7f95f406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.786+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f960c088d50 con 0x7f96140b28b0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.786+0000 7f9613fff640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f95f406ae40 0x7f95f406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:32.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.787+0000 7f9613fff640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f95f406ae40 0x7f95f406d300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f95fc007d40 tx=0x7f95fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:32.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.790+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f960c08c030 con 0x7f96140b28b0 2023-12-16T21:15:32.967 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.968+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9614069400 con 0x7f96140b28b0 2023-12-16T21:15:32.968 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.969+0000 7f9610ff9640 1 -- 172.21.15.46:0/122088816 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f960c053eb0 con 0x7f96140b28b0 2023-12-16T21:15:32.969 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:32.969 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":49,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:31.993111+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:32.970 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f95f406ae40 msgr2=0x7f95f406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f95f406ae40 0x7f95f406d300 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f95fc007d40 tx=0x7f95fc009040 comp rx=0 tx=0).stop 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 msgr2=0x7f961415c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 0x7f961415c240 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f961406e5c0 tx=0x7f960c00fd40 comp rx=0 tx=0).stop 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 shutdown_connections 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f95f406ae40 0x7f95f406d300 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f96140b28b0 0x7f961415c240 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96141548c0 0x7f96140b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 --2- 172.21.15.46:0/122088816 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9600003f20 0x7f96140b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 >> 172.21.15.46:0/122088816 conn(0x7f961406a4d0 msgr2=0x7f961406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 shutdown_connections 2023-12-16T21:15:32.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:32.971+0000 7f961a548640 1 -- 172.21.15.46:0/122088816 wait complete. 2023-12-16T21:15:32.982 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:33.088 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.089+0000 7f8bde809640 1 Processor -- start 2023-12-16T21:15:33.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.089+0000 7f8bde809640 1 -- start start 2023-12-16T21:15:33.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bde809640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8bd80b1400 0x7f8bd815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bde809640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd815e2a0 con 0x7f8bd8143490 2023-12-16T21:15:33.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bde809640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd815b240 con 0x7f8bd815e730 2023-12-16T21:15:33.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bde809640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd815b520 con 0x7f8bd80b1400 2023-12-16T21:15:33.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f8bd8143490 0x7f8bd80b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34430/0 (socket says 172.21.15.46:34430) 2023-12-16T21:15:33.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd7fff640 1 -- 172.21.15.46:0/1619608802 learned_addr learned my addr 172.21.15.46:0/1619608802 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd77fe640 1 --2- 172.21.15.46:0/1619608802 >> v2:172.21.15.121:3300/0 conn(0x7f8bd80b1400 0x7f8bd815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774495701 0 0) 0x7f8bd815e2a0 con 0x7f8bd8143490 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc4003520 con 0x7f8bd8143490 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143869748 0 0) 0x7f8bd815b240 con 0x7f8bd815e730 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bd815e2a0 con 0x7f8bd815e730 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1195632690 0 0) 0x7f8bc4003520 con 0x7f8bd8143490 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd77fe640 1 -- 172.21.15.46:0/1619608802 >> v1:172.21.15.121:6789/0 conn(0x7f8bd815e730 legacy=0x7f8bd815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.090+0000 7f8bd77fe640 1 -- 172.21.15.46:0/1619608802 >> v1:172.21.15.46:6789/0 conn(0x7f8bd8143490 legacy=0x7f8bd80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.091 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.091+0000 7f8bd77fe640 1 -- 172.21.15.46:0/1619608802 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd815dc50 con 0x7f8bd80b1400 2023-12-16T21:15:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.091+0000 7f8bd77fe640 1 --2- 172.21.15.46:0/1619608802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd80b1400 0x7f8bd815a4e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00097a0 tx=0x7f8bc002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd183ad44f1f3234 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.091+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc002f050 con 0x7f8bd80b1400 2023-12-16T21:15:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc0031500 con 0x7f8bd80b1400 2023-12-16T21:15:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bd6ffd640 1 -- 172.21.15.46:0/1619608802 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8001670 con 0x7f8bd80b1400 2023-12-16T21:15:33.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 -- 172.21.15.46:0/1619608802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd80b1400 msgr2=0x7f8bd815a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 --2- 172.21.15.46:0/1619608802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd80b1400 0x7f8bd815a4e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00097a0 tx=0x7f8bc002b820 comp rx=0 tx=0).stop 2023-12-16T21:15:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 -- 172.21.15.46:0/1619608802 shutdown_connections 2023-12-16T21:15:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 --2- 172.21.15.46:0/1619608802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd80b1400 0x7f8bd815a4e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 -- 172.21.15.46:0/1619608802 >> 172.21.15.46:0/1619608802 conn(0x7f8bd806a4d0 msgr2=0x7f8bd806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 -- 172.21.15.46:0/1619608802 shutdown_connections 2023-12-16T21:15:33.093 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.092+0000 7f8bde809640 1 -- 172.21.15.46:0/1619608802 wait complete. 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.093+0000 7f8bde809640 1 Processor -- start 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.093+0000 7f8bde809640 1 -- start start 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bde809640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 0x7f8bd81427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bde809640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd8143490 0x7f8bd8139a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bde809640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd815e730 0x7f8bd8159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bde809640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd813bf60 con 0x7f8bd80b1400 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bde809640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd8142e20 con 0x7f8bd8143490 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bde809640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bd80aeba0 con 0x7f8bd815e730 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 0x7f8bd81427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 0x7f8bd81427b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59664/0 (socket says 172.21.15.46:59664) 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bd7fff640 1 -- 172.21.15.46:0/2333557234 learned_addr learned my addr 172.21.15.46:0/2333557234 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.094+0000 7f8bd77fe640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd8143490 0x7f8bd8139a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bdcda6640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd815e730 0x7f8bd8159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd7fff640 1 -- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd815e730 msgr2=0x7f8bd8159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd7fff640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd815e730 0x7f8bd8159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd7fff640 1 -- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd8143490 msgr2=0x7f8bd8139a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd7fff640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd8143490 0x7f8bd8139a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd7fff640 1 -- 172.21.15.46:0/2333557234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc0009400 con 0x7f8bd80b1400 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd77fe640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd8143490 0x7f8bd8139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd7fff640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 0x7f8bd81427b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800c710 tx=0x7f8bc800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8005ad0 con 0x7f8bd80b1400 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc8005c70 con 0x7f8bd80b1400 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8007070 con 0x7f8bd80b1400 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd80aee80 con 0x7f8bd80b1400 2023-12-16T21:15:33.096 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.095+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd80af440 con 0x7f8bd80b1400 2023-12-16T21:15:33.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.096+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd8069400 con 0x7f8bd80b1400 2023-12-16T21:15:33.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.096+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8bc80071d0 con 0x7f8bd80b1400 2023-12-16T21:15:33.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.097+0000 7f8bd4ff9640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8bb806ae40 0x7f8bb806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.097+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bc8088c50 con 0x7f8bd80b1400 2023-12-16T21:15:33.097 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.097+0000 7f8bd77fe640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8bb806ae40 0x7f8bb806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.098 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.098+0000 7f8bd77fe640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8bb806ae40 0x7f8bb806d300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0035000 tx=0x7f8bc0008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.100 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.101+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8bc808e020 con 0x7f8bd80b1400 2023-12-16T21:15:33.277 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.278+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bd806b4e0 con 0x7f8bd80b1400 2023-12-16T21:15:33.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.279+0000 7f8bd4ff9640 1 -- 172.21.15.46:0/2333557234 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16582 (secure 0 0 0) 0x7f8bc8017020 con 0x7f8bd80b1400 2023-12-16T21:15:33.279 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:33.279 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":49,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:31.993111+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8bb806ae40 msgr2=0x7f8bb806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8bb806ae40 0x7f8bb806d300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0035000 tx=0x7f8bc0008030 comp rx=0 tx=0).stop 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 msgr2=0x7f8bd81427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 0x7f8bd81427b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800c710 tx=0x7f8bc800cbd0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 shutdown_connections 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8bb806ae40 0x7f8bb806d300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8bd815e730 0x7f8bd8159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bd8143490 0x7f8bd8139a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.281 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 --2- 172.21.15.46:0/2333557234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8bd80b1400 0x7f8bd81427b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 >> 172.21.15.46:0/2333557234 conn(0x7f8bd806a4d0 msgr2=0x7f8bd8070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:33.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 shutdown_connections 2023-12-16T21:15:33.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.281+0000 7f8bde809640 1 -- 172.21.15.46:0/2333557234 wait complete. 2023-12-16T21:15:33.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-16T21:15:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.402+0000 7f44dce09640 1 Processor -- start 2023-12-16T21:15:33.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.402+0000 7f44dce09640 1 -- start start 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.403+0000 7f44dce09640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44d815d330 0x7f44d80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.403+0000 7f44dce09640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d2910 0x7f44d80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.403+0000 7f44dce09640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44d80b2b10 con 0x7f44d80d3730 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.403+0000 7f44dce09640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44d80d3b50 con 0x7f44d815d330 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.403+0000 7f44dce09640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80d1590 con 0x7f44d80d2910 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d6d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f44d80d3730 0x7f44d80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34440/0 (socket says 172.21.15.46:34440) 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d6d9d640 1 -- 172.21.15.46:0/3554904664 learned_addr learned my addr 172.21.15.46:0/3554904664 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d5d9b640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d2910 0x7f44d80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d659c640 1 --2- 172.21.15.46:0/3554904664 >> v2:172.21.15.121:3300/0 conn(0x7f44d815d330 0x7f44d80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d559a640 1 -- 172.21.15.46:0/3554904664 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671179804 0 0) 0x7f44d80b2b10 con 0x7f44d80d3730 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d559a640 1 -- 172.21.15.46:0/3554904664 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c0003520 con 0x7f44d80d3730 2023-12-16T21:15:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d659c640 1 -- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d2910 msgr2=0x7f44d80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d659c640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d2910 0x7f44d80d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d659c640 1 -- 172.21.15.46:0/3554904664 >> v1:172.21.15.46:6789/0 conn(0x7f44d80d3730 legacy=0x7f44d80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d659c640 1 -- 172.21.15.46:0/3554904664 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d80d1870 con 0x7f44d815d330 2023-12-16T21:15:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.404+0000 7f44d5d9b640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d2910 0x7f44d80d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.405+0000 7f44d659c640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d815d330 0x7f44d80b8b80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f44c80097a0 tx=0x7f44c802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36dd0876f73a12f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.405+0000 7f44d559a640 1 -- 172.21.15.46:0/3554904664 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c802e070 con 0x7f44d815d330 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.405+0000 7f44d559a640 1 -- 172.21.15.46:0/3554904664 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44c80315a0 con 0x7f44d815d330 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.405+0000 7f44d559a640 1 -- 172.21.15.46:0/3554904664 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c8031910 con 0x7f44d815d330 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.405+0000 7f44dce09640 1 -- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d815d330 msgr2=0x7f44d80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.405+0000 7f44dce09640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d815d330 0x7f44d80b8b80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f44c80097a0 tx=0x7f44c802b840 comp rx=0 tx=0).stop 2023-12-16T21:15:33.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.406+0000 7f44dce09640 1 -- 172.21.15.46:0/3554904664 shutdown_connections 2023-12-16T21:15:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.406+0000 7f44dce09640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d2910 0x7f44d80d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.406+0000 7f44dce09640 1 --2- 172.21.15.46:0/3554904664 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d815d330 0x7f44d80b8b80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.406+0000 7f44dce09640 1 -- 172.21.15.46:0/3554904664 >> 172.21.15.46:0/3554904664 conn(0x7f44d8072020 msgr2=0x7f44d8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.406+0000 7f44dce09640 1 -- 172.21.15.46:0/3554904664 shutdown_connections 2023-12-16T21:15:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.406+0000 7f44dce09640 1 -- 172.21.15.46:0/3554904664 wait complete. 2023-12-16T21:15:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.407+0000 7f44dce09640 1 Processor -- start 2023-12-16T21:15:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.407+0000 7f44dce09640 1 -- start start 2023-12-16T21:15:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44dce09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d80d2910 0x7f44d80cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44dce09640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d3730 0x7f44d80c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44dce09640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 0x7f44d81f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44dce09640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d815d710 con 0x7f44d815d330 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44dce09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d80cb0a0 con 0x7f44d80d2910 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44dce09640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44d81f6bc0 con 0x7f44d80d3730 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 0x7f44d81f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 0x7f44d81f5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59672/0 (socket says 172.21.15.46:59672) 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 -- 172.21.15.46:0/4181327772 learned_addr learned my addr 172.21.15.46:0/4181327772 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d5d9b640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d3730 0x7f44d80c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d659c640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d80d2910 0x7f44d80cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 -- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d3730 msgr2=0x7f44d80c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d3730 0x7f44d80c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 -- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d80d2910 msgr2=0x7f44d80cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d80d2910 0x7f44d80cab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 -- 172.21.15.46:0/4181327772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44cc009400 con 0x7f44d815d330 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d659c640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d80d2910 0x7f44d80cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44d6d9d640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 0x7f44d81f5ed0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f44d000cb40 tx=0x7f44d000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.408+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d000aca0 con 0x7f44d815d330 2023-12-16T21:15:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.409+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d000ae00 con 0x7f44d815d330 2023-12-16T21:15:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.409+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0012620 con 0x7f44d815d330 2023-12-16T21:15:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.409+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c8009400 con 0x7f44d815d330 2023-12-16T21:15:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.409+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d815da10 con 0x7f44d815d330 2023-12-16T21:15:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.410+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8069400 con 0x7f44d815d330 2023-12-16T21:15:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.410+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f44d0007070 con 0x7f44d815d330 2023-12-16T21:15:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.411+0000 7f44c6ffd640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44b806ae40 0x7f44b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.411+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44d0089680 con 0x7f44d815d330 2023-12-16T21:15:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.411+0000 7f44d659c640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44b806ae40 0x7f44b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.412+0000 7f44d659c640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44b806ae40 0x7f44b806d300 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f44c8036000 tx=0x7f44c803d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.414 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.415+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f44d0054720 con 0x7f44d815d330 2023-12-16T21:15:33.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2023-12-16T21:15:33.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.627+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f44d80b64d0 con 0x7f44d815d330 2023-12-16T21:15:33.636 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.637+0000 7f44c6ffd640 1 -- 172.21.15.46:0/4181327772 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v19) v1 ==== 118+0+0 (secure 0 0 0) 0x7f44d0057d40 con 0x7f44d815d330 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.638+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44b806ae40 msgr2=0x7f44b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.638+0000 7f44dce09640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44b806ae40 0x7f44b806d300 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f44c8036000 tx=0x7f44c803d040 comp rx=0 tx=0).stop 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 msgr2=0x7f44d81f5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 0x7f44d81f5ed0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f44d000cb40 tx=0x7f44d000a060 comp rx=0 tx=0).stop 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 shutdown_connections 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f44b806ae40 0x7f44b806d300 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44d815d330 0x7f44d81f5ed0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44d80d3730 0x7f44d80c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 --2- 172.21.15.46:0/4181327772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44d80d2910 0x7f44d80cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 >> 172.21.15.46:0/4181327772 conn(0x7f44d8072020 msgr2=0x7f44d8072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 shutdown_connections 2023-12-16T21:15:33.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.639+0000 7f44dce09640 1 -- 172.21.15.46:0/4181327772 wait complete. 2023-12-16T21:15:33.650 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-16T21:15:33.650 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-16T21:15:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.760+0000 7fbab5cc9640 1 Processor -- start 2023-12-16T21:15:33.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.760+0000 7fbab5cc9640 1 -- start start 2023-12-16T21:15:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbab5cc9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbab006ea50 0x7fbab006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbab5cc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab00d0ed0 0x7fbab00d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbab5cc9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab0161d20 con 0x7fbab0160fb0 2023-12-16T21:15:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbab5cc9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab006f370 con 0x7fbab006ea50 2023-12-16T21:15:33.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbab5cc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab015e760 con 0x7fbab00d0ed0 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbaaffff640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab00d0ed0 0x7fbab00d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbaaf7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbab006ea50 0x7fbab006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbaaeffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbab0160fb0 0x7fbab0161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57176/0 (socket says 172.21.15.46:57176) 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.761+0000 7fbaaeffd640 1 -- 172.21.15.46:0/2108687578 learned_addr learned my addr 172.21.15.46:0/2108687578 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.762+0000 7fbaaffff640 1 -- 172.21.15.46:0/2108687578 >> v2:172.21.15.121:3300/0 conn(0x7fbab006ea50 msgr2=0x7fbab006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.762+0000 7fbaaffff640 1 --2- 172.21.15.46:0/2108687578 >> v2:172.21.15.121:3300/0 conn(0x7fbab006ea50 0x7fbab006ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.762+0000 7fbaaffff640 1 -- 172.21.15.46:0/2108687578 >> v1:172.21.15.121:6789/0 conn(0x7fbab0160fb0 legacy=0x7fbab0161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.762+0000 7fbaaffff640 1 -- 172.21.15.46:0/2108687578 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab015d9a0 con 0x7fbab00d0ed0 2023-12-16T21:15:33.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.762+0000 7fbaaf7fe640 1 --2- 172.21.15.46:0/2108687578 >> v2:172.21.15.121:3300/0 conn(0x7fbab006ea50 0x7fbab006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.762+0000 7fbaaffff640 1 --2- 172.21.15.46:0/2108687578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab00d0ed0 0x7fbab00d1320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009d90 tx=0x7fbaa802bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b114eefde65bd241 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbaae7fc640 1 -- 172.21.15.46:0/2108687578 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa802ca10 con 0x7fbab00d0ed0 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbaae7fc640 1 -- 172.21.15.46:0/2108687578 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa802cb70 con 0x7fbab00d0ed0 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbaae7fc640 1 -- 172.21.15.46:0/2108687578 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8034440 con 0x7fbab00d0ed0 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 -- 172.21.15.46:0/2108687578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab00d0ed0 msgr2=0x7fbab00d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/2108687578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab00d0ed0 0x7fbab00d1320 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009d90 tx=0x7fbaa802bce0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 -- 172.21.15.46:0/2108687578 shutdown_connections 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/2108687578 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab00d0ed0 0x7fbab00d1320 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/2108687578 >> v2:172.21.15.121:3300/0 conn(0x7fbab006ea50 0x7fbab006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 -- 172.21.15.46:0/2108687578 >> 172.21.15.46:0/2108687578 conn(0x7fbab006a6b0 msgr2=0x7fbab006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:33.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 -- 172.21.15.46:0/2108687578 shutdown_connections 2023-12-16T21:15:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.763+0000 7fbab5cc9640 1 -- 172.21.15.46:0/2108687578 wait complete. 2023-12-16T21:15:33.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.764+0000 7fbab5cc9640 1 Processor -- start 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 -- start start 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab006ea50 0x7fbab0156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbab00d0ed0 0x7fbab01e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 0x7fbab012f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab013f8f0 con 0x7fbab0160fb0 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0156eb0 con 0x7fbab00d0ed0 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbab5cc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0158210 con 0x7fbab006ea50 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbaaffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 0x7fbab012f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbaaffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 0x7fbab012f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59674/0 (socket says 172.21.15.46:59674) 2023-12-16T21:15:33.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.765+0000 7fbaaffff640 1 -- 172.21.15.46:0/1012958849 learned_addr learned my addr 172.21.15.46:0/1012958849 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaf7fe640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab006ea50 0x7fbab0156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaeffd640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbab00d0ed0 0x7fbab01e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaffff640 1 -- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab006ea50 msgr2=0x7fbab0156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaffff640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab006ea50 0x7fbab0156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaffff640 1 -- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbab00d0ed0 msgr2=0x7fbab01e7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaffff640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbab00d0ed0 0x7fbab01e7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaffff640 1 -- 172.21.15.46:0/1012958849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4009400 con 0x7fbab0160fb0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbaaffff640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 0x7fbab012f440 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009d90 tx=0x7fbaa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8040650 con 0x7fbab0160fb0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa80407b0 con 0x7fbab0160fb0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa8009a70 con 0x7fbab0160fb0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8040b60 con 0x7fbab0160fb0 2023-12-16T21:15:33.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.766+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab01587d0 con 0x7fbab0160fb0 2023-12-16T21:15:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.767+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab015b260 con 0x7fbab0160fb0 2023-12-16T21:15:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.768+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbaa8040cc0 con 0x7fbab0160fb0 2023-12-16T21:15:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.768+0000 7fba97fff640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba9c06ae40 0x7fba9c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.768+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbaa80ac160 con 0x7fbab0160fb0 2023-12-16T21:15:33.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.768+0000 7fbaaf7fe640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba9c06ae40 0x7fba9c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:33.770 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.769+0000 7fbaaf7fe640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba9c06ae40 0x7fba9c06d300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009c30 tx=0x7fbaa4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:33.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.772+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fbaa80772c0 con 0x7fbab0160fb0 2023-12-16T21:15:33.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:33.985+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbab0069400 con 0x7fbab0160fb0 2023-12-16T21:15:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.648+0000 7fba97fff640 1 -- 172.21.15.46:0/1012958849 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbaa8009450 con 0x7fbab0160fb0 2023-12-16T21:15:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.650+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba9c06ae40 msgr2=0x7fba9c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.650+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba9c06ae40 0x7fba9c06d300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4009c30 tx=0x7fbaa4009190 comp rx=0 tx=0).stop 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.650+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 msgr2=0x7fbab012f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.650+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 0x7fbab012f440 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009d90 tx=0x7fbaa8004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 shutdown_connections 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba9c06ae40 0x7fba9c06d300 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbab0160fb0 0x7fbab012f440 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbab00d0ed0 0x7fbab01e7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 --2- 172.21.15.46:0/1012958849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbab006ea50 0x7fbab0156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 >> 172.21.15.46:0/1012958849 conn(0x7fbab006a6b0 msgr2=0x7fbab01416e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 shutdown_connections 2023-12-16T21:15:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.651+0000 7fbab5cc9640 1 -- 172.21.15.46:0/1012958849 wait complete. 2023-12-16T21:15:34.664 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-16T21:15:34.664 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:34.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 Processor -- start 2023-12-16T21:15:34.776 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 -- start start 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca0015d330 0x7fca000b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000d3900 0x7fca000b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca000b2b10 con 0x7fca000d2910 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca000d1590 con 0x7fca0015d330 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.777+0000 7fca0650f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca000d1870 con 0x7fca000d3900 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca0015d330 0x7fca000b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fca0015d330 0x7fca000b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59688/0 (socket says 172.21.15.46:59688) 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 -- 172.21.15.46:0/1620166282 learned_addr learned my addr 172.21.15.46:0/1620166282 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fca04aac640 1 --2- 172.21.15.46:0/1620166282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000d3900 0x7fca000b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 -- 172.21.15.46:0/1620166282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000d3900 msgr2=0x7fca000b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 --2- 172.21.15.46:0/1620166282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000d3900 0x7fca000b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.778 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 -- 172.21.15.46:0/1620166282 >> v1:172.21.15.46:6789/0 conn(0x7fca000d2910 legacy=0x7fca000d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:34.779 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.778+0000 7fc9fffff640 1 -- 172.21.15.46:0/1620166282 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca000d1b50 con 0x7fca0015d330 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fc9fffff640 1 --2- 172.21.15.46:0/1620166282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca0015d330 0x7fca000b8b80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009870 tx=0x7fc9f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb1c5df2da70ed1 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fc9feffd640 1 -- 172.21.15.46:0/1620166282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0031640 con 0x7fca0015d330 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fc9feffd640 1 -- 172.21.15.46:0/1620166282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f00317a0 con 0x7fca0015d330 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fc9feffd640 1 -- 172.21.15.46:0/1620166282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f0031b30 con 0x7fca0015d330 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fca0650f640 1 -- 172.21.15.46:0/1620166282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca0015d330 msgr2=0x7fca000b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fca0650f640 1 --2- 172.21.15.46:0/1620166282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca0015d330 0x7fca000b8b80 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0009870 tx=0x7fc9f002b8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fca0650f640 1 -- 172.21.15.46:0/1620166282 shutdown_connections 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fca0650f640 1 --2- 172.21.15.46:0/1620166282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca000d3900 0x7fca000b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fca0650f640 1 --2- 172.21.15.46:0/1620166282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca0015d330 0x7fca000b8b80 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.779+0000 7fca0650f640 1 -- 172.21.15.46:0/1620166282 >> 172.21.15.46:0/1620166282 conn(0x7fca00072020 msgr2=0x7fca00072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.780+0000 7fca0650f640 1 -- 172.21.15.46:0/1620166282 shutdown_connections 2023-12-16T21:15:34.780 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.780+0000 7fca0650f640 1 -- 172.21.15.46:0/1620166282 wait complete. 2023-12-16T21:15:34.782 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.781+0000 7fca0650f640 1 Processor -- start 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.781+0000 7fca0650f640 1 -- start start 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000d2910 0x7fca0015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 0x7fca000cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca0015d330 0x7fca000ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000cd320 con 0x7fca000d3900 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca000cb440 con 0x7fca000d2910 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0015ebc0 con 0x7fca0015d330 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 0x7fca000cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:34.783 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 0x7fca000cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59700/0 (socket says 172.21.15.46:59700) 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 -- 172.21.15.46:0/2547042802 learned_addr learned my addr 172.21.15.46:0/2547042802 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9fffff640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000d2910 0x7fca0015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca04aac640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca0015d330 0x7fca000ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 -- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca0015d330 msgr2=0x7fca000ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca0015d330 0x7fca000ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 -- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000d2910 msgr2=0x7fca0015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000d2910 0x7fca0015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 -- 172.21.15.46:0/2547042802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f00094d0 con 0x7fca000d3900 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca04aac640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca0015d330 0x7fca000ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9fffff640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000d2910 0x7fca0015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9ff7fe640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 0x7fca000cc8a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400c400 tx=0x7fc9f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40052d0 con 0x7fca000d3900 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f4005430 con 0x7fca000d3900 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca000cae80 con 0x7fca000d3900 2023-12-16T21:15:34.784 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.782+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4003830 con 0x7fca000d3900 2023-12-16T21:15:34.785 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.783+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca000ca4d0 con 0x7fca000d3900 2023-12-16T21:15:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.784+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca000ca310 con 0x7fca000d3900 2023-12-16T21:15:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.784+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc9f4003990 con 0x7fca000d3900 2023-12-16T21:15:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.785+0000 7fc9fcff9640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9ec06ae40 0x7fc9ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.785+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9f4088c20 con 0x7fca000d3900 2023-12-16T21:15:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.785+0000 7fc9fffff640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9ec06ae40 0x7fc9ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:34.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.785+0000 7fc9fffff640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9ec06ae40 0x7fc9ec06d300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0036000 tx=0x7fc9f003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:34.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:34.789+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc9f4053c30 con 0x7fca000d3900 2023-12-16T21:15:34.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.000+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca00069400 con 0x7fca000d3900 2023-12-16T21:15:35.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.000+0000 7fc9fcff9640 1 -- 172.21.15.46:0/2547042802 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fc9f4017090 con 0x7fca000d3900 2023-12-16T21:15:35.000 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:35.000 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:15:31.993483+0000","modified":"2023-12-16T21:15:33.986432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"b","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/2015861199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":2015861199},{"type":"v1","addr":"172.21.15.121:6837","nonce":2015861199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-16T21:15:35.001 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9ec06ae40 msgr2=0x7fc9ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9ec06ae40 0x7fc9ec06d300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc9f0036000 tx=0x7fc9f003c040 comp rx=0 tx=0).stop 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 msgr2=0x7fca000cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 0x7fca000cc8a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400c400 tx=0x7fc9f400c8c0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 shutdown_connections 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc9ec06ae40 0x7fc9ec06d300 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca000d3900 0x7fca000cc8a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca0015d330 0x7fca000ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 --2- 172.21.15.46:0/2547042802 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca000d2910 0x7fca0015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 >> 172.21.15.46:0/2547042802 conn(0x7fca00072020 msgr2=0x7fca000d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 shutdown_connections 2023-12-16T21:15:35.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.003+0000 7fca0650f640 1 -- 172.21.15.46:0/2547042802 wait complete. 2023-12-16T21:15:35.014 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 Processor -- start 2023-12-16T21:15:35.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 -- start start 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f3958161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f39580d0ed0 0x7f39580d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3958161d20 con 0x7f395806ea50 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f395815e760 con 0x7f39580d0ed0 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.121+0000 7f395eae7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395815d9a0 con 0x7f3958160fb0 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f395c883640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f395806ea50 0x7f395806ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34450/0 (socket says 172.21.15.46:34450) 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f395c883640 1 -- 172.21.15.46:0/2294636130 learned_addr learned my addr 172.21.15.46:0/2294636130 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f3957fff640 1 --2- 172.21.15.46:0/2294636130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f3958161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f395d084640 1 --2- 172.21.15.46:0/2294636130 >> v2:172.21.15.121:3300/0 conn(0x7f39580d0ed0 0x7f39580d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f39577fe640 1 -- 172.21.15.46:0/2294636130 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1869878481 0 0) 0x7f3958161d20 con 0x7f395806ea50 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f39577fe640 1 -- 172.21.15.46:0/2294636130 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3948003520 con 0x7f395806ea50 2023-12-16T21:15:35.122 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f3957fff640 1 -- 172.21.15.46:0/2294636130 >> v2:172.21.15.121:3300/0 conn(0x7f39580d0ed0 msgr2=0x7f39580d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f3957fff640 1 --2- 172.21.15.46:0/2294636130 >> v2:172.21.15.121:3300/0 conn(0x7f39580d0ed0 0x7f39580d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f3957fff640 1 -- 172.21.15.46:0/2294636130 >> v1:172.21.15.46:6789/0 conn(0x7f395806ea50 legacy=0x7f395806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f39577fe640 1 -- 172.21.15.46:0/2294636130 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941534823 0 0) 0x7f3948003520 con 0x7f395806ea50 2023-12-16T21:15:35.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.122+0000 7f3957fff640 1 -- 172.21.15.46:0/2294636130 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395815dc80 con 0x7f3958160fb0 2023-12-16T21:15:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.123+0000 7f3957fff640 1 --2- 172.21.15.46:0/2294636130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f3958161440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f39440097a0 tx=0x7f394402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5488864ccc319691 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.123+0000 7f39577fe640 1 -- 172.21.15.46:0/2294636130 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c001670 con 0x7f3958160fb0 2023-12-16T21:15:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.123+0000 7f39577fe640 1 -- 172.21.15.46:0/2294636130 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394402e050 con 0x7f3958160fb0 2023-12-16T21:15:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.123+0000 7f39577fe640 1 -- 172.21.15.46:0/2294636130 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944030730 con 0x7f3958160fb0 2023-12-16T21:15:35.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.123+0000 7f395eae7640 1 -- 172.21.15.46:0/2294636130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 msgr2=0x7f3958161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.123+0000 7f395eae7640 1 --2- 172.21.15.46:0/2294636130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f3958161440 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f39440097a0 tx=0x7f394402b790 comp rx=0 tx=0).stop 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.124+0000 7f395eae7640 1 -- 172.21.15.46:0/2294636130 shutdown_connections 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.124+0000 7f395eae7640 1 --2- 172.21.15.46:0/2294636130 >> v2:172.21.15.121:3300/0 conn(0x7f39580d0ed0 0x7f39580d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.124+0000 7f395eae7640 1 --2- 172.21.15.46:0/2294636130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f3958161440 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.124+0000 7f395eae7640 1 -- 172.21.15.46:0/2294636130 >> 172.21.15.46:0/2294636130 conn(0x7f395806a6b0 msgr2=0x7f395806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.124+0000 7f395eae7640 1 -- 172.21.15.46:0/2294636130 shutdown_connections 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.124+0000 7f395eae7640 1 -- 172.21.15.46:0/2294636130 wait complete. 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.125+0000 7f395eae7640 1 Processor -- start 2023-12-16T21:15:35.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.125+0000 7f395eae7640 1 -- start start 2023-12-16T21:15:35.127 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 0x7f39581e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39580d0ed0 0x7f3958156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f395815f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39581e8ef0 con 0x7f395806ea50 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3958157ff0 con 0x7f39580d0ed0 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39581582d0 con 0x7f3958160fb0 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 0x7f39581e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 0x7f39581e8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59706/0 (socket says 172.21.15.46:59706) 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 -- 172.21.15.46:0/1756219665 learned_addr learned my addr 172.21.15.46:0/1756219665 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f3957fff640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39580d0ed0 0x7f3958156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 -- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 msgr2=0x7f395815f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f395815f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 -- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39580d0ed0 msgr2=0x7f3958156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39580d0ed0 0x7f3958156080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.128 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 -- 172.21.15.46:0/1756219665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3944009400 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395c883640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 0x7f39581e8610 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f394c00c700 tx=0x7f394c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c005ad0 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f394c005c30 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39581585b0 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394c007070 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.126+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39581e8b50 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.127+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3958158ad0 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.128+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f394c0071d0 con 0x7f395806ea50 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.128+0000 7f39557fa640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f393c06ae40 0x7f393c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.128+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f394c089260 con 0x7f395806ea50 2023-12-16T21:15:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.128+0000 7f3957fff640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f393c06ae40 0x7f393c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.129+0000 7f3957fff640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f393c06ae40 0x7f393c06d300 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3944034000 tx=0x7f394402f000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.132+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3958158ad0 con 0x7f395806ea50 2023-12-16T21:15:35.309 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.309+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3958158ad0 con 0x7f395806ea50 2023-12-16T21:15:35.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.310+0000 7f39557fa640 1 -- 172.21.15.46:0/1756219665 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f3958158ad0 con 0x7f395806ea50 2023-12-16T21:15:35.311 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:35.311 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f393c06ae40 msgr2=0x7f393c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f393c06ae40 0x7f393c06d300 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3944034000 tx=0x7f394402f000 comp rx=0 tx=0).stop 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 msgr2=0x7f39581e8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 0x7f39581e8610 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f394c00c700 tx=0x7f394c00cbc0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 shutdown_connections 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f393c06ae40 0x7f393c06d300 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3958160fb0 0x7f395815f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39580d0ed0 0x7f3958156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 --2- 172.21.15.46:0/1756219665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f395806ea50 0x7f39581e8610 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 >> 172.21.15.46:0/1756219665 conn(0x7f395806a6b0 msgr2=0x7f395806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 shutdown_connections 2023-12-16T21:15:35.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.312+0000 7f395eae7640 1 -- 172.21.15.46:0/1756219665 wait complete. 2023-12-16T21:15:35.323 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-16T21:15:35.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:15:35.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.429+0000 7fd863ee0640 1 Processor -- start 2023-12-16T21:15:35.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.430+0000 7fd863ee0640 1 -- start start 2023-12-16T21:15:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.430+0000 7fd863ee0640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd85c15d1d0 0x7fd85c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.430+0000 7fd863ee0640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd85c0ad640 0x7fd85c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.430+0000 7fd863ee0640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85c0adae0 con 0x7fd85c0aeb80 2023-12-16T21:15:35.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd863ee0640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85c0addc0 con 0x7fd85c0ad640 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd863ee0640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85c0ae0a0 con 0x7fd85c15d1d0 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd85c0ad640 0x7fd85c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd85c0ad640 0x7fd85c06ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59722/0 (socket says 172.21.15.46:59722) 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 -- 172.21.15.46:0/1175110630 learned_addr learned my addr 172.21.15.46:0/1175110630 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd861c7c640 1 --2- 172.21.15.46:0/1175110630 >> v2:172.21.15.121:3300/0 conn(0x7fd85c15d1d0 0x7fd85c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 -- 172.21.15.46:0/1175110630 >> v2:172.21.15.121:3300/0 conn(0x7fd85c15d1d0 msgr2=0x7fd85c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 --2- 172.21.15.46:0/1175110630 >> v2:172.21.15.121:3300/0 conn(0x7fd85c15d1d0 0x7fd85c06b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 -- 172.21.15.46:0/1175110630 >> v1:172.21.15.46:6789/0 conn(0x7fd85c0aeb80 legacy=0x7fd85c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd86147b640 1 -- 172.21.15.46:0/1175110630 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c0ae380 con 0x7fd85c0ad640 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.431+0000 7fd861c7c640 1 --2- 172.21.15.46:0/1175110630 >> v2:172.21.15.121:3300/0 conn(0x7fd85c15d1d0 0x7fd85c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd86147b640 1 --2- 172.21.15.46:0/1175110630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c0ad640 0x7fd85c06ba20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd85c072310 tx=0x7fd84c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15afb8be30553a77 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd860c7a640 1 -- 172.21.15.46:0/1175110630 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c030860 con 0x7fd85c0ad640 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd860c7a640 1 -- 172.21.15.46:0/1175110630 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd84c0309c0 con 0x7fd85c0ad640 2023-12-16T21:15:35.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd860c7a640 1 -- 172.21.15.46:0/1175110630 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd84c030d50 con 0x7fd85c0ad640 2023-12-16T21:15:35.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd863ee0640 1 -- 172.21.15.46:0/1175110630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c0ad640 msgr2=0x7fd85c06ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1175110630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c0ad640 0x7fd85c06ba20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd85c072310 tx=0x7fd84c02b9a0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd863ee0640 1 -- 172.21.15.46:0/1175110630 shutdown_connections 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1175110630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c0ad640 0x7fd85c06ba20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1175110630 >> v2:172.21.15.121:3300/0 conn(0x7fd85c15d1d0 0x7fd85c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.432+0000 7fd863ee0640 1 -- 172.21.15.46:0/1175110630 >> 172.21.15.46:0/1175110630 conn(0x7fd85c070e70 msgr2=0x7fd85c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.433+0000 7fd863ee0640 1 -- 172.21.15.46:0/1175110630 shutdown_connections 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.433+0000 7fd863ee0640 1 -- 172.21.15.46:0/1175110630 wait complete. 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 Processor -- start 2023-12-16T21:15:35.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 -- start start 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd85c0ad640 0x7fd85c158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd85c0aeb80 0x7fd85c158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 0x7fd85c15fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c1597f0 con 0x7fd85c15d1d0 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c159ad0 con 0x7fd85c0ad640 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.434+0000 7fd863ee0640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c159db0 con 0x7fd85c0aeb80 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 0x7fd85c15fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 0x7fd85c15fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59734/0 (socket says 172.21.15.46:59734) 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 -- 172.21.15.46:0/1374186087 learned_addr learned my addr 172.21.15.46:0/1374186087 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86147b640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd85c0aeb80 0x7fd85c158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd861c7c640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd85c0ad640 0x7fd85c158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 -- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd85c0aeb80 msgr2=0x7fd85c158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd85c0aeb80 0x7fd85c158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 -- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd85c0ad640 msgr2=0x7fd85c158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd85c0ad640 0x7fd85c158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 -- 172.21.15.46:0/1374186087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84c0094d0 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd861c7c640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd85c0ad640 0x7fd85c158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd86247d640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 0x7fd85c15fcb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c7a0 tx=0x7fd85400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854003b70 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd854003cf0 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c15db00 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854003ea0 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.435+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c149410 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.436+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c15dcf0 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.437+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd85400aa90 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.437+0000 7fd8527fc640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd84006ae40 0x7fd84006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.437+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd854089430 con 0x7fd85c15d1d0 2023-12-16T21:15:35.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.437+0000 7fd861c7c640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd84006ae40 0x7fd84006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.438+0000 7fd861c7c640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd84006ae40 0x7fd84006d300 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd858006fd0 tx=0x7fd858008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.441+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd854054450 con 0x7fd85c15d1d0 2023-12-16T21:15:35.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.616+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd85c0b2b10 con 0x7fd84006ae40 2023-12-16T21:15:35.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.624+0000 7fd8527fc640 1 -- 172.21.15.46:0/1374186087 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289405 (secure 0 0 0) 0x7fd858003e90 con 0x7fd84006ae40 2023-12-16T21:15:35.627 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:35.629 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:15:35.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd84006ae40 msgr2=0x7fd84006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.630 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd84006ae40 0x7fd84006d300 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd858006fd0 tx=0x7fd858008040 comp rx=0 tx=0).stop 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 msgr2=0x7fd85c15fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 0x7fd85c15fcb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c7a0 tx=0x7fd85400cc60 comp rx=0 tx=0).stop 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 shutdown_connections 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd84006ae40 0x7fd84006d300 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd85c15d1d0 0x7fd85c15fcb0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd85c0aeb80 0x7fd85c158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 --2- 172.21.15.46:0/1374186087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd85c0ad640 0x7fd85c158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.627+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 >> 172.21.15.46:0/1374186087 conn(0x7fd85c070e70 msgr2=0x7fd85c06d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.628+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 shutdown_connections 2023-12-16T21:15:35.631 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.628+0000 7fd863ee0640 1 -- 172.21.15.46:0/1374186087 wait complete. 2023-12-16T21:15:35.638 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2023-12-16T21:15:33.983433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238520,"kb_used_data":5680,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736200,"statfs":{"total":773094113280,"available":772849868800,"internally_reserved":0,"allocated":5816320,"data_stored":3078046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000831"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648563+0000","last_change":"2023-12-16T21:15:32.642665+0000","last_active":"2023-12-16T21:15:32.648563+0000","last_peered":"2023-12-16T21:15:32.648563+0000","last_clean":"2023-12-16T21:15:32.648563+0000","last_became_active":"2023-12-16T21:15:32.642356+0000","last_became_peered":"2023-12-16T21:15:32.642356+0000","last_unstale":"2023-12-16T21:15:32.648563+0000","last_undegraded":"2023-12-16T21:15:32.648563+0000","last_fullsized":"2023-12-16T21:15:32.648563+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:47:10.770226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654866+0000","last_change":"2023-12-16T21:15:32.651177+0000","last_active":"2023-12-16T21:15:32.654866+0000","last_peered":"2023-12-16T21:15:32.654866+0000","last_clean":"2023-12-16T21:15:32.654866+0000","last_became_active":"2023-12-16T21:15:32.650714+0000","last_became_peered":"2023-12-16T21:15:32.650714+0000","last_unstale":"2023-12-16T21:15:32.654866+0000","last_undegraded":"2023-12-16T21:15:32.654866+0000","last_fullsized":"2023-12-16T21:15:32.654866+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654479+0000","last_change":"2023-12-16T21:15:32.646414+0000","last_active":"2023-12-16T21:15:32.654479+0000","last_peered":"2023-12-16T21:15:32.654479+0000","last_clean":"2023-12-16T21:15:32.654479+0000","last_became_active":"2023-12-16T21:15:32.646132+0000","last_became_peered":"2023-12-16T21:15:32.646132+0000","last_unstale":"2023-12-16T21:15:32.654479+0000","last_undegraded":"2023-12-16T21:15:32.654479+0000","last_fullsized":"2023-12-16T21:15:32.654479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:25:43.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654680+0000","last_change":"2023-12-16T21:15:32.650750+0000","last_active":"2023-12-16T21:15:32.654680+0000","last_peered":"2023-12-16T21:15:32.654680+0000","last_clean":"2023-12-16T21:15:32.654680+0000","last_became_active":"2023-12-16T21:15:32.650470+0000","last_became_peered":"2023-12-16T21:15:32.650470+0000","last_unstale":"2023-12-16T21:15:32.654680+0000","last_undegraded":"2023-12-16T21:15:32.654680+0000","last_fullsized":"2023-12-16T21:15:32.654680+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:35.587180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654925+0000","last_change":"2023-12-16T21:15:32.652762+0000","last_active":"2023-12-16T21:15:32.654925+0000","last_peered":"2023-12-16T21:15:32.654925+0000","last_clean":"2023-12-16T21:15:32.654925+0000","last_became_active":"2023-12-16T21:15:32.652505+0000","last_became_peered":"2023-12-16T21:15:32.652505+0000","last_unstale":"2023-12-16T21:15:32.654925+0000","last_undegraded":"2023-12-16T21:15:32.654925+0000","last_fullsized":"2023-12-16T21:15:32.654925+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.646166+0000","last_change":"2023-12-16T21:15:32.639650+0000","last_active":"2023-12-16T21:15:32.646166+0000","last_peered":"2023-12-16T21:15:32.646166+0000","last_clean":"2023-12-16T21:15:32.646166+0000","last_became_active":"2023-12-16T21:15:32.639354+0000","last_became_peered":"2023-12-16T21:15:32.639354+0000","last_unstale":"2023-12-16T21:15:32.646166+0000","last_undegraded":"2023-12-16T21:15:32.646166+0000","last_fullsized":"2023-12-16T21:15:32.646166+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647215+0000","last_change":"2023-12-16T21:15:32.644479+0000","last_active":"2023-12-16T21:15:32.647215+0000","last_peered":"2023-12-16T21:15:32.647215+0000","last_clean":"2023-12-16T21:15:32.647215+0000","last_became_active":"2023-12-16T21:15:32.644048+0000","last_became_peered":"2023-12-16T21:15:32.644048+0000","last_unstale":"2023-12-16T21:15:32.647215+0000","last_undegraded":"2023-12-16T21:15:32.647215+0000","last_fullsized":"2023-12-16T21:15:32.647215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:02:52.674851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659252+0000","last_change":"2023-12-16T21:15:32.651612+0000","last_active":"2023-12-16T21:15:32.659252+0000","last_peered":"2023-12-16T21:15:32.659252+0000","last_clean":"2023-12-16T21:15:32.659252+0000","last_became_active":"2023-12-16T21:15:32.651310+0000","last_became_peered":"2023-12-16T21:15:32.651310+0000","last_unstale":"2023-12-16T21:15:32.659252+0000","last_undegraded":"2023-12-16T21:15:32.659252+0000","last_fullsized":"2023-12-16T21:15:32.659252+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:52.341272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658218+0000","last_change":"2023-12-16T21:15:32.651032+0000","last_active":"2023-12-16T21:15:32.658218+0000","last_peered":"2023-12-16T21:15:32.658218+0000","last_clean":"2023-12-16T21:15:32.658218+0000","last_became_active":"2023-12-16T21:15:32.650633+0000","last_became_peered":"2023-12-16T21:15:32.650633+0000","last_unstale":"2023-12-16T21:15:32.658218+0000","last_undegraded":"2023-12-16T21:15:32.658218+0000","last_fullsized":"2023-12-16T21:15:32.658218+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:26:37.791962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.646131+0000","last_change":"2023-12-16T21:15:32.637193+0000","last_active":"2023-12-16T21:15:32.646131+0000","last_peered":"2023-12-16T21:15:32.646131+0000","last_clean":"2023-12-16T21:15:32.646131+0000","last_became_active":"2023-12-16T21:15:32.636914+0000","last_became_peered":"2023-12-16T21:15:32.636914+0000","last_unstale":"2023-12-16T21:15:32.646131+0000","last_undegraded":"2023-12-16T21:15:32.646131+0000","last_fullsized":"2023-12-16T21:15:32.646131+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:14:01.565210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644927+0000","last_change":"2023-12-16T21:15:32.641734+0000","last_active":"2023-12-16T21:15:32.644927+0000","last_peered":"2023-12-16T21:15:32.644927+0000","last_clean":"2023-12-16T21:15:32.644927+0000","last_became_active":"2023-12-16T21:15:32.641452+0000","last_became_peered":"2023-12-16T21:15:32.641452+0000","last_unstale":"2023-12-16T21:15:32.644927+0000","last_undegraded":"2023-12-16T21:15:32.644927+0000","last_fullsized":"2023-12-16T21:15:32.644927+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:36:59.166199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658968+0000","last_change":"2023-12-16T21:15:32.656392+0000","last_active":"2023-12-16T21:15:32.658968+0000","last_peered":"2023-12-16T21:15:32.658968+0000","last_clean":"2023-12-16T21:15:32.658968+0000","last_became_active":"2023-12-16T21:15:32.655344+0000","last_became_peered":"2023-12-16T21:15:32.655344+0000","last_unstale":"2023-12-16T21:15:32.658968+0000","last_undegraded":"2023-12-16T21:15:32.658968+0000","last_fullsized":"2023-12-16T21:15:32.658968+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:15:38.975571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644680+0000","last_change":"2023-12-16T21:15:32.637168+0000","last_active":"2023-12-16T21:15:32.644680+0000","last_peered":"2023-12-16T21:15:32.644680+0000","last_clean":"2023-12-16T21:15:32.644680+0000","last_became_active":"2023-12-16T21:15:32.636875+0000","last_became_peered":"2023-12-16T21:15:32.636875+0000","last_unstale":"2023-12-16T21:15:32.644680+0000","last_undegraded":"2023-12-16T21:15:32.644680+0000","last_fullsized":"2023-12-16T21:15:32.644680+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:16.640850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647341+0000","last_change":"2023-12-16T21:15:32.644616+0000","last_active":"2023-12-16T21:15:32.647341+0000","last_peered":"2023-12-16T21:15:32.647341+0000","last_clean":"2023-12-16T21:15:32.647341+0000","last_became_active":"2023-12-16T21:15:32.644324+0000","last_became_peered":"2023-12-16T21:15:32.644324+0000","last_unstale":"2023-12-16T21:15:32.647341+0000","last_undegraded":"2023-12-16T21:15:32.647341+0000","last_fullsized":"2023-12-16T21:15:32.647341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:16.640850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654557+0000","last_change":"2023-12-16T21:15:32.650698+0000","last_active":"2023-12-16T21:15:32.654557+0000","last_peered":"2023-12-16T21:15:32.654557+0000","last_clean":"2023-12-16T21:15:32.654557+0000","last_became_active":"2023-12-16T21:15:32.650389+0000","last_became_peered":"2023-12-16T21:15:32.650389+0000","last_unstale":"2023-12-16T21:15:32.654557+0000","last_undegraded":"2023-12-16T21:15:32.654557+0000","last_fullsized":"2023-12-16T21:15:32.654557+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:58:58.752236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648198+0000","last_change":"2023-12-16T21:15:32.641683+0000","last_active":"2023-12-16T21:15:32.648198+0000","last_peered":"2023-12-16T21:15:32.648198+0000","last_clean":"2023-12-16T21:15:32.648198+0000","last_became_active":"2023-12-16T21:15:32.641389+0000","last_became_peered":"2023-12-16T21:15:32.641389+0000","last_unstale":"2023-12-16T21:15:32.648198+0000","last_undegraded":"2023-12-16T21:15:32.648198+0000","last_fullsized":"2023-12-16T21:15:32.648198+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647655+0000","last_change":"2023-12-16T21:15:32.642697+0000","last_active":"2023-12-16T21:15:32.647655+0000","last_peered":"2023-12-16T21:15:32.647655+0000","last_clean":"2023-12-16T21:15:32.647655+0000","last_became_active":"2023-12-16T21:15:32.642400+0000","last_became_peered":"2023-12-16T21:15:32.642400+0000","last_unstale":"2023-12-16T21:15:32.647655+0000","last_undegraded":"2023-12-16T21:15:32.647655+0000","last_fullsized":"2023-12-16T21:15:32.647655+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:39.167149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659671+0000","last_change":"2023-12-16T21:15:32.653851+0000","last_active":"2023-12-16T21:15:32.659671+0000","last_peered":"2023-12-16T21:15:32.659671+0000","last_clean":"2023-12-16T21:15:32.659671+0000","last_became_active":"2023-12-16T21:15:32.653538+0000","last_became_peered":"2023-12-16T21:15:32.653538+0000","last_unstale":"2023-12-16T21:15:32.659671+0000","last_undegraded":"2023-12-16T21:15:32.659671+0000","last_fullsized":"2023-12-16T21:15:32.659671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:32:58.884843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.646552+0000","last_change":"2023-12-16T21:15:32.642522+0000","last_active":"2023-12-16T21:15:32.646552+0000","last_peered":"2023-12-16T21:15:32.646552+0000","last_clean":"2023-12-16T21:15:32.646552+0000","last_became_active":"2023-12-16T21:15:32.642213+0000","last_became_peered":"2023-12-16T21:15:32.642213+0000","last_unstale":"2023-12-16T21:15:32.646552+0000","last_undegraded":"2023-12-16T21:15:32.646552+0000","last_fullsized":"2023-12-16T21:15:32.646552+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:47:10.770226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657418+0000","last_change":"2023-12-16T21:15:32.652127+0000","last_active":"2023-12-16T21:15:32.657418+0000","last_peered":"2023-12-16T21:15:32.657418+0000","last_clean":"2023-12-16T21:15:32.657418+0000","last_became_active":"2023-12-16T21:15:32.651856+0000","last_became_peered":"2023-12-16T21:15:32.651856+0000","last_unstale":"2023-12-16T21:15:32.657418+0000","last_undegraded":"2023-12-16T21:15:32.657418+0000","last_fullsized":"2023-12-16T21:15:32.657418+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:25:44.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657363+0000","last_change":"2023-12-16T21:15:32.652066+0000","last_active":"2023-12-16T21:15:32.657363+0000","last_peered":"2023-12-16T21:15:32.657363+0000","last_clean":"2023-12-16T21:15:32.657363+0000","last_became_active":"2023-12-16T21:15:32.651792+0000","last_became_peered":"2023-12-16T21:15:32.651792+0000","last_unstale":"2023-12-16T21:15:32.657363+0000","last_undegraded":"2023-12-16T21:15:32.657363+0000","last_fullsized":"2023-12-16T21:15:32.657363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:39.668395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658325+0000","last_change":"2023-12-16T21:15:32.651318+0000","last_active":"2023-12-16T21:15:32.658325+0000","last_peered":"2023-12-16T21:15:32.658325+0000","last_clean":"2023-12-16T21:15:32.658325+0000","last_became_active":"2023-12-16T21:15:32.651011+0000","last_became_peered":"2023-12-16T21:15:32.651011+0000","last_unstale":"2023-12-16T21:15:32.658325+0000","last_undegraded":"2023-12-16T21:15:32.658325+0000","last_fullsized":"2023-12-16T21:15:32.658325+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:36:43.705381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644995+0000","last_change":"2023-12-16T21:15:32.641932+0000","last_active":"2023-12-16T21:15:32.644995+0000","last_peered":"2023-12-16T21:15:32.644995+0000","last_clean":"2023-12-16T21:15:32.644995+0000","last_became_active":"2023-12-16T21:15:32.641542+0000","last_became_peered":"2023-12-16T21:15:32.641542+0000","last_unstale":"2023-12-16T21:15:32.644995+0000","last_undegraded":"2023-12-16T21:15:32.644995+0000","last_fullsized":"2023-12-16T21:15:32.644995+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:16:33.086634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.637974+0000","last_change":"2023-12-16T21:15:32.637346+0000","last_active":"2023-12-16T21:15:32.637974+0000","last_peered":"2023-12-16T21:15:32.637974+0000","last_clean":"2023-12-16T21:15:32.637974+0000","last_became_active":"2023-12-16T21:15:32.637073+0000","last_became_peered":"2023-12-16T21:15:32.637073+0000","last_unstale":"2023-12-16T21:15:32.637974+0000","last_undegraded":"2023-12-16T21:15:32.637974+0000","last_fullsized":"2023-12-16T21:15:32.637974+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:17:27.143791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656159+0000","last_change":"2023-12-16T21:15:32.648955+0000","last_active":"2023-12-16T21:15:32.656159+0000","last_peered":"2023-12-16T21:15:32.656159+0000","last_clean":"2023-12-16T21:15:32.656159+0000","last_became_active":"2023-12-16T21:15:32.647872+0000","last_became_peered":"2023-12-16T21:15:32.647872+0000","last_unstale":"2023-12-16T21:15:32.656159+0000","last_undegraded":"2023-12-16T21:15:32.656159+0000","last_fullsized":"2023-12-16T21:15:32.656159+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657928+0000","last_change":"2023-12-16T21:15:32.650294+0000","last_active":"2023-12-16T21:15:32.657928+0000","last_peered":"2023-12-16T21:15:32.657928+0000","last_clean":"2023-12-16T21:15:32.657928+0000","last_became_active":"2023-12-16T21:15:32.650029+0000","last_became_peered":"2023-12-16T21:15:32.650029+0000","last_unstale":"2023-12-16T21:15:32.657928+0000","last_undegraded":"2023-12-16T21:15:32.657928+0000","last_fullsized":"2023-12-16T21:15:32.657928+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:51:15.168522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648880+0000","last_change":"2023-12-16T21:15:32.642557+0000","last_active":"2023-12-16T21:15:32.648880+0000","last_peered":"2023-12-16T21:15:32.648880+0000","last_clean":"2023-12-16T21:15:32.648880+0000","last_became_active":"2023-12-16T21:15:32.642284+0000","last_became_peered":"2023-12-16T21:15:32.642284+0000","last_unstale":"2023-12-16T21:15:32.648880+0000","last_undegraded":"2023-12-16T21:15:32.648880+0000","last_fullsized":"2023-12-16T21:15:32.648880+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:43.155045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.649080+0000","last_change":"2023-12-16T21:15:32.646191+0000","last_active":"2023-12-16T21:15:32.649080+0000","last_peered":"2023-12-16T21:15:32.649080+0000","last_clean":"2023-12-16T21:15:32.649080+0000","last_became_active":"2023-12-16T21:15:32.645731+0000","last_became_peered":"2023-12-16T21:15:32.645731+0000","last_unstale":"2023-12-16T21:15:32.649080+0000","last_undegraded":"2023-12-16T21:15:32.649080+0000","last_fullsized":"2023-12-16T21:15:32.649080+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:41:31.484631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656191+0000","last_change":"2023-12-16T21:15:32.648907+0000","last_active":"2023-12-16T21:15:32.656191+0000","last_peered":"2023-12-16T21:15:32.656191+0000","last_clean":"2023-12-16T21:15:32.656191+0000","last_became_active":"2023-12-16T21:15:32.648614+0000","last_became_peered":"2023-12-16T21:15:32.648614+0000","last_unstale":"2023-12-16T21:15:32.656191+0000","last_undegraded":"2023-12-16T21:15:32.656191+0000","last_fullsized":"2023-12-16T21:15:32.656191+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:54:35.100969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.650716+0000","last_change":"2023-12-16T21:15:32.646961+0000","last_active":"2023-12-16T21:15:32.650716+0000","last_peered":"2023-12-16T21:15:32.650716+0000","last_clean":"2023-12-16T21:15:32.650716+0000","last_became_active":"2023-12-16T21:15:32.646680+0000","last_became_peered":"2023-12-16T21:15:32.646680+0000","last_unstale":"2023-12-16T21:15:32.650716+0000","last_undegraded":"2023-12-16T21:15:32.650716+0000","last_fullsized":"2023-12-16T21:15:32.650716+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:43.155045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647671+0000","last_change":"2023-12-16T21:15:32.645008+0000","last_active":"2023-12-16T21:15:32.647671+0000","last_peered":"2023-12-16T21:15:32.647671+0000","last_clean":"2023-12-16T21:15:32.647671+0000","last_became_active":"2023-12-16T21:15:32.643968+0000","last_became_peered":"2023-12-16T21:15:32.643968+0000","last_unstale":"2023-12-16T21:15:32.647671+0000","last_undegraded":"2023-12-16T21:15:32.647671+0000","last_fullsized":"2023-12-16T21:15:32.647671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:28:33.936466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.651276+0000","last_change":"2023-12-16T21:15:32.647529+0000","last_active":"2023-12-16T21:15:32.651276+0000","last_peered":"2023-12-16T21:15:32.651276+0000","last_clean":"2023-12-16T21:15:32.651276+0000","last_became_active":"2023-12-16T21:15:32.647260+0000","last_became_peered":"2023-12-16T21:15:32.647260+0000","last_unstale":"2023-12-16T21:15:32.651276+0000","last_undegraded":"2023-12-16T21:15:32.651276+0000","last_fullsized":"2023-12-16T21:15:32.651276+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659483+0000","last_change":"2023-12-16T21:15:32.652204+0000","last_active":"2023-12-16T21:15:32.659483+0000","last_peered":"2023-12-16T21:15:32.659483+0000","last_clean":"2023-12-16T21:15:32.659483+0000","last_became_active":"2023-12-16T21:15:32.651925+0000","last_became_peered":"2023-12-16T21:15:32.651925+0000","last_unstale":"2023-12-16T21:15:32.659483+0000","last_undegraded":"2023-12-16T21:15:32.659483+0000","last_fullsized":"2023-12-16T21:15:32.659483+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":15,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657510+0000","last_change":"2023-12-16T21:15:31.634592+0000","last_active":"2023-12-16T21:15:32.657510+0000","last_peered":"2023-12-16T21:15:32.657510+0000","last_clean":"2023-12-16T21:15:32.657510+0000","last_became_active":"2023-12-16T21:15:31.634363+0000","last_became_peered":"2023-12-16T21:15:31.634363+0000","last_unstale":"2023-12-16T21:15:32.657510+0000","last_undegraded":"2023-12-16T21:15:32.657510+0000","last_fullsized":"2023-12-16T21:15:32.657510+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:46.114548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644340+0000","last_change":"2023-12-16T21:15:32.641959+0000","last_active":"2023-12-16T21:15:32.644340+0000","last_peered":"2023-12-16T21:15:32.644340+0000","last_clean":"2023-12-16T21:15:32.644340+0000","last_became_active":"2023-12-16T21:15:32.641396+0000","last_became_peered":"2023-12-16T21:15:32.641396+0000","last_unstale":"2023-12-16T21:15:32.644340+0000","last_undegraded":"2023-12-16T21:15:32.644340+0000","last_fullsized":"2023-12-16T21:15:32.644340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658733+0000","last_change":"2023-12-16T21:15:31.643125+0000","last_active":"2023-12-16T21:15:32.658733+0000","last_peered":"2023-12-16T21:15:32.658733+0000","last_clean":"2023-12-16T21:15:32.658733+0000","last_became_active":"2023-12-16T21:15:31.642850+0000","last_became_peered":"2023-12-16T21:15:31.642850+0000","last_unstale":"2023-12-16T21:15:32.658733+0000","last_undegraded":"2023-12-16T21:15:32.658733+0000","last_fullsized":"2023-12-16T21:15:32.658733+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:25:42.211225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657492+0000","last_change":"2023-12-16T21:15:32.655542+0000","last_active":"2023-12-16T21:15:32.657492+0000","last_peered":"2023-12-16T21:15:32.657492+0000","last_clean":"2023-12-16T21:15:32.657492+0000","last_became_active":"2023-12-16T21:15:32.655222+0000","last_became_peered":"2023-12-16T21:15:32.655222+0000","last_unstale":"2023-12-16T21:15:32.657492+0000","last_undegraded":"2023-12-16T21:15:32.657492+0000","last_fullsized":"2023-12-16T21:15:32.657492+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:18:38.476202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657500+0000","last_change":"2023-12-16T21:15:31.636839+0000","last_active":"2023-12-16T21:15:32.657500+0000","last_peered":"2023-12-16T21:15:32.657500+0000","last_clean":"2023-12-16T21:15:32.657500+0000","last_became_active":"2023-12-16T21:15:31.635892+0000","last_became_peered":"2023-12-16T21:15:31.635892+0000","last_unstale":"2023-12-16T21:15:32.657500+0000","last_undegraded":"2023-12-16T21:15:32.657500+0000","last_fullsized":"2023-12-16T21:15:32.657500+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:46.990678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657820+0000","last_change":"2023-12-16T21:15:32.648190+0000","last_active":"2023-12-16T21:15:32.657820+0000","last_peered":"2023-12-16T21:15:32.657820+0000","last_clean":"2023-12-16T21:15:32.657820+0000","last_became_active":"2023-12-16T21:15:32.647878+0000","last_became_peered":"2023-12-16T21:15:32.647878+0000","last_unstale":"2023-12-16T21:15:32.657820+0000","last_undegraded":"2023-12-16T21:15:32.657820+0000","last_fullsized":"2023-12-16T21:15:32.657820+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:06:32.894407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659261+0000","last_change":"2023-12-16T21:15:31.637718+0000","last_active":"2023-12-16T21:15:32.659261+0000","last_peered":"2023-12-16T21:15:32.659261+0000","last_clean":"2023-12-16T21:15:32.659261+0000","last_became_active":"2023-12-16T21:15:31.637445+0000","last_became_peered":"2023-12-16T21:15:31.637445+0000","last_unstale":"2023-12-16T21:15:32.659261+0000","last_undegraded":"2023-12-16T21:15:32.659261+0000","last_fullsized":"2023-12-16T21:15:32.659261+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:32:57.874803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654260+0000","last_change":"2023-12-16T21:15:32.647424+0000","last_active":"2023-12-16T21:15:32.654260+0000","last_peered":"2023-12-16T21:15:32.654260+0000","last_clean":"2023-12-16T21:15:32.654260+0000","last_became_active":"2023-12-16T21:15:32.647149+0000","last_became_peered":"2023-12-16T21:15:32.647149+0000","last_unstale":"2023-12-16T21:15:32.654260+0000","last_undegraded":"2023-12-16T21:15:32.654260+0000","last_fullsized":"2023-12-16T21:15:32.654260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659393+0000","last_change":"2023-12-16T21:15:31.638127+0000","last_active":"2023-12-16T21:15:32.659393+0000","last_peered":"2023-12-16T21:15:32.659393+0000","last_clean":"2023-12-16T21:15:32.659393+0000","last_became_active":"2023-12-16T21:15:31.637844+0000","last_became_peered":"2023-12-16T21:15:31.637844+0000","last_unstale":"2023-12-16T21:15:32.659393+0000","last_undegraded":"2023-12-16T21:15:32.659393+0000","last_fullsized":"2023-12-16T21:15:32.659393+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:20:51.013316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.651641+0000","last_change":"2023-12-16T21:15:32.648186+0000","last_active":"2023-12-16T21:15:32.651641+0000","last_peered":"2023-12-16T21:15:32.651641+0000","last_clean":"2023-12-16T21:15:32.651641+0000","last_became_active":"2023-12-16T21:15:32.647924+0000","last_became_peered":"2023-12-16T21:15:32.647924+0000","last_unstale":"2023-12-16T21:15:32.651641+0000","last_undegraded":"2023-12-16T21:15:32.651641+0000","last_fullsized":"2023-12-16T21:15:32.651641+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:48.000718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.650155+0000","last_change":"2023-12-16T21:15:31.633365+0000","last_active":"2023-12-16T21:15:32.650155+0000","last_peered":"2023-12-16T21:15:32.650155+0000","last_clean":"2023-12-16T21:15:32.650155+0000","last_became_active":"2023-12-16T21:15:31.633097+0000","last_became_peered":"2023-12-16T21:15:31.633097+0000","last_unstale":"2023-12-16T21:15:32.650155+0000","last_undegraded":"2023-12-16T21:15:32.650155+0000","last_fullsized":"2023-12-16T21:15:32.650155+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:14:00.555170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657867+0000","last_change":"2023-12-16T21:15:32.648913+0000","last_active":"2023-12-16T21:15:32.657867+0000","last_peered":"2023-12-16T21:15:32.657867+0000","last_clean":"2023-12-16T21:15:32.657867+0000","last_became_active":"2023-12-16T21:15:32.648611+0000","last_became_peered":"2023-12-16T21:15:32.648611+0000","last_unstale":"2023-12-16T21:15:32.657867+0000","last_undegraded":"2023-12-16T21:15:32.657867+0000","last_fullsized":"2023-12-16T21:15:32.657867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:38:12.849334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.640878+0000","last_change":"2023-12-16T21:15:31.635059+0000","last_active":"2023-12-16T21:15:32.640878+0000","last_peered":"2023-12-16T21:15:32.640878+0000","last_clean":"2023-12-16T21:15:32.640878+0000","last_became_active":"2023-12-16T21:15:31.634198+0000","last_became_peered":"2023-12-16T21:15:31.634198+0000","last_unstale":"2023-12-16T21:15:32.640878+0000","last_undegraded":"2023-12-16T21:15:32.640878+0000","last_fullsized":"2023-12-16T21:15:32.640878+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:21:23.853508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654687+0000","last_change":"2023-12-16T21:15:32.651533+0000","last_active":"2023-12-16T21:15:32.654687+0000","last_peered":"2023-12-16T21:15:32.654687+0000","last_clean":"2023-12-16T21:15:32.654687+0000","last_became_active":"2023-12-16T21:15:32.651119+0000","last_became_peered":"2023-12-16T21:15:32.651119+0000","last_unstale":"2023-12-16T21:15:32.654687+0000","last_undegraded":"2023-12-16T21:15:32.654687+0000","last_fullsized":"2023-12-16T21:15:32.654687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:29:51.206182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.629648+0000","last_change":"2023-12-16T21:15:31.634129+0000","last_active":"2023-12-16T21:15:32.629648+0000","last_peered":"2023-12-16T21:15:32.629648+0000","last_clean":"2023-12-16T21:15:32.629648+0000","last_became_active":"2023-12-16T21:15:31.633849+0000","last_became_peered":"2023-12-16T21:15:31.633849+0000","last_unstale":"2023-12-16T21:15:32.629648+0000","last_undegraded":"2023-12-16T21:15:32.629648+0000","last_fullsized":"2023-12-16T21:15:32.629648+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:43.310583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654215+0000","last_change":"2023-12-16T21:15:32.645960+0000","last_active":"2023-12-16T21:15:32.654215+0000","last_peered":"2023-12-16T21:15:32.654215+0000","last_clean":"2023-12-16T21:15:32.654215+0000","last_became_active":"2023-12-16T21:15:32.645705+0000","last_became_peered":"2023-12-16T21:15:32.645705+0000","last_unstale":"2023-12-16T21:15:32.654215+0000","last_undegraded":"2023-12-16T21:15:32.654215+0000","last_fullsized":"2023-12-16T21:15:32.654215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:54:35.100969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.636327+0000","last_change":"2023-12-16T21:15:31.636895+0000","last_active":"2023-12-16T21:15:32.636327+0000","last_peered":"2023-12-16T21:15:32.636327+0000","last_clean":"2023-12-16T21:15:32.636327+0000","last_became_active":"2023-12-16T21:15:31.636634+0000","last_became_peered":"2023-12-16T21:15:31.636634+0000","last_unstale":"2023-12-16T21:15:32.636327+0000","last_undegraded":"2023-12-16T21:15:32.636327+0000","last_fullsized":"2023-12-16T21:15:32.636327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:25:29.874049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.652590+0000","last_change":"2023-12-16T21:15:32.650407+0000","last_active":"2023-12-16T21:15:32.652590+0000","last_peered":"2023-12-16T21:15:32.652590+0000","last_clean":"2023-12-16T21:15:32.652590+0000","last_became_active":"2023-12-16T21:15:32.649946+0000","last_became_peered":"2023-12-16T21:15:32.649946+0000","last_unstale":"2023-12-16T21:15:32.652590+0000","last_undegraded":"2023-12-16T21:15:32.652590+0000","last_fullsized":"2023-12-16T21:15:32.652590+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:47:10.770226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.633754+0000","last_change":"2023-12-16T21:15:31.639976+0000","last_active":"2023-12-16T21:15:32.633754+0000","last_peered":"2023-12-16T21:15:32.633754+0000","last_clean":"2023-12-16T21:15:32.633754+0000","last_became_active":"2023-12-16T21:15:31.639673+0000","last_became_peered":"2023-12-16T21:15:31.639673+0000","last_unstale":"2023-12-16T21:15:32.633754+0000","last_undegraded":"2023-12-16T21:15:32.633754+0000","last_fullsized":"2023-12-16T21:15:32.633754+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:38.157109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657013+0000","last_change":"2023-12-16T21:15:32.648595+0000","last_active":"2023-12-16T21:15:32.657013+0000","last_peered":"2023-12-16T21:15:32.657013+0000","last_clean":"2023-12-16T21:15:32.657013+0000","last_became_active":"2023-12-16T21:15:32.648311+0000","last_became_peered":"2023-12-16T21:15:32.648311+0000","last_unstale":"2023-12-16T21:15:32.657013+0000","last_undegraded":"2023-12-16T21:15:32.657013+0000","last_fullsized":"2023-12-16T21:15:32.657013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656327+0000","last_change":"2023-12-16T21:15:31.641593+0000","last_active":"2023-12-16T21:15:32.656327+0000","last_peered":"2023-12-16T21:15:32.656327+0000","last_clean":"2023-12-16T21:15:32.656327+0000","last_became_active":"2023-12-16T21:15:31.641248+0000","last_became_peered":"2023-12-16T21:15:31.641248+0000","last_unstale":"2023-12-16T21:15:32.656327+0000","last_undegraded":"2023-12-16T21:15:32.656327+0000","last_fullsized":"2023-12-16T21:15:32.656327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:40:03.528345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654301+0000","last_change":"2023-12-16T21:15:32.646803+0000","last_active":"2023-12-16T21:15:32.654301+0000","last_peered":"2023-12-16T21:15:32.654301+0000","last_clean":"2023-12-16T21:15:32.654301+0000","last_became_active":"2023-12-16T21:15:32.646493+0000","last_became_peered":"2023-12-16T21:15:32.646493+0000","last_unstale":"2023-12-16T21:15:32.654301+0000","last_undegraded":"2023-12-16T21:15:32.654301+0000","last_fullsized":"2023-12-16T21:15:32.654301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:16.640850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":17,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.660889+0000","last_change":"2023-12-16T21:15:31.625494+0000","last_active":"2023-12-16T21:15:32.660889+0000","last_peered":"2023-12-16T21:15:32.660889+0000","last_clean":"2023-12-16T21:15:32.660889+0000","last_became_active":"2023-12-16T21:15:31.625179+0000","last_became_peered":"2023-12-16T21:15:31.625179+0000","last_unstale":"2023-12-16T21:15:32.660889+0000","last_undegraded":"2023-12-16T21:15:32.660889+0000","last_fullsized":"2023-12-16T21:15:32.660889+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:49:44.106454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657446+0000","last_change":"2023-12-16T21:15:32.655931+0000","last_active":"2023-12-16T21:15:32.657446+0000","last_peered":"2023-12-16T21:15:32.657446+0000","last_clean":"2023-12-16T21:15:32.657446+0000","last_became_active":"2023-12-16T21:15:32.655628+0000","last_became_peered":"2023-12-16T21:15:32.655628+0000","last_unstale":"2023-12-16T21:15:32.657446+0000","last_undegraded":"2023-12-16T21:15:32.657446+0000","last_fullsized":"2023-12-16T21:15:32.657446+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:48.000718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.660240+0000","last_change":"2023-12-16T21:15:31.633030+0000","last_active":"2023-12-16T21:15:32.660240+0000","last_peered":"2023-12-16T21:15:32.660240+0000","last_clean":"2023-12-16T21:15:32.660240+0000","last_became_active":"2023-12-16T21:15:31.632728+0000","last_became_peered":"2023-12-16T21:15:31.632728+0000","last_unstale":"2023-12-16T21:15:32.660240+0000","last_undegraded":"2023-12-16T21:15:32.660240+0000","last_fullsized":"2023-12-16T21:15:32.660240+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:40:03.528345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644735+0000","last_change":"2023-12-16T21:15:32.640797+0000","last_active":"2023-12-16T21:15:32.644735+0000","last_peered":"2023-12-16T21:15:32.644735+0000","last_clean":"2023-12-16T21:15:32.644735+0000","last_became_active":"2023-12-16T21:15:32.640523+0000","last_became_peered":"2023-12-16T21:15:32.640523+0000","last_unstale":"2023-12-16T21:15:32.644735+0000","last_undegraded":"2023-12-16T21:15:32.644735+0000","last_fullsized":"2023-12-16T21:15:32.644735+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:58:58.752236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.640401+0000","last_change":"2023-12-16T21:15:31.635005+0000","last_active":"2023-12-16T21:15:32.640401+0000","last_peered":"2023-12-16T21:15:32.640401+0000","last_clean":"2023-12-16T21:15:32.640401+0000","last_became_active":"2023-12-16T21:15:31.634153+0000","last_became_peered":"2023-12-16T21:15:31.634153+0000","last_unstale":"2023-12-16T21:15:32.640401+0000","last_undegraded":"2023-12-16T21:15:32.640401+0000","last_fullsized":"2023-12-16T21:15:32.640401+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:44:27.335742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658477+0000","last_change":"2023-12-16T21:15:32.652820+0000","last_active":"2023-12-16T21:15:32.658477+0000","last_peered":"2023-12-16T21:15:32.658477+0000","last_clean":"2023-12-16T21:15:32.658477+0000","last_became_active":"2023-12-16T21:15:32.652314+0000","last_became_peered":"2023-12-16T21:15:32.652314+0000","last_unstale":"2023-12-16T21:15:32.658477+0000","last_undegraded":"2023-12-16T21:15:32.658477+0000","last_fullsized":"2023-12-16T21:15:32.658477+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:53:53.952000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.636461+0000","last_change":"2023-12-16T21:15:31.636870+0000","last_active":"2023-12-16T21:15:32.636461+0000","last_peered":"2023-12-16T21:15:32.636461+0000","last_clean":"2023-12-16T21:15:32.636461+0000","last_became_active":"2023-12-16T21:15:31.636576+0000","last_became_peered":"2023-12-16T21:15:31.636576+0000","last_unstale":"2023-12-16T21:15:32.636461+0000","last_undegraded":"2023-12-16T21:15:32.636461+0000","last_fullsized":"2023-12-16T21:15:32.636461+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:04.601140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647633+0000","last_change":"2023-12-16T21:15:32.642005+0000","last_active":"2023-12-16T21:15:32.647633+0000","last_peered":"2023-12-16T21:15:32.647633+0000","last_clean":"2023-12-16T21:15:32.647633+0000","last_became_active":"2023-12-16T21:15:32.641745+0000","last_became_peered":"2023-12-16T21:15:32.641745+0000","last_unstale":"2023-12-16T21:15:32.647633+0000","last_undegraded":"2023-12-16T21:15:32.647633+0000","last_fullsized":"2023-12-16T21:15:32.647633+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":19,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659176+0000","last_change":"2023-12-16T21:15:31.631537+0000","last_active":"2023-12-16T21:15:32.659176+0000","last_peered":"2023-12-16T21:15:32.659176+0000","last_clean":"2023-12-16T21:15:32.659176+0000","last_became_active":"2023-12-16T21:15:31.631249+0000","last_became_peered":"2023-12-16T21:15:31.631249+0000","last_unstale":"2023-12-16T21:15:32.659176+0000","last_undegraded":"2023-12-16T21:15:32.659176+0000","last_fullsized":"2023-12-16T21:15:32.659176+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:32:30.221788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659265+0000","last_change":"2023-12-16T21:15:32.653334+0000","last_active":"2023-12-16T21:15:32.659265+0000","last_peered":"2023-12-16T21:15:32.659265+0000","last_clean":"2023-12-16T21:15:32.659265+0000","last_became_active":"2023-12-16T21:15:32.653088+0000","last_became_peered":"2023-12-16T21:15:32.653088+0000","last_unstale":"2023-12-16T21:15:32.659265+0000","last_undegraded":"2023-12-16T21:15:32.659265+0000","last_fullsized":"2023-12-16T21:15:32.659265+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:05.611180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658997+0000","last_change":"2023-12-16T21:15:31.638956+0000","last_active":"2023-12-16T21:15:32.658997+0000","last_peered":"2023-12-16T21:15:32.658997+0000","last_clean":"2023-12-16T21:15:32.658997+0000","last_became_active":"2023-12-16T21:15:31.638657+0000","last_became_peered":"2023-12-16T21:15:31.638657+0000","last_unstale":"2023-12-16T21:15:32.658997+0000","last_undegraded":"2023-12-16T21:15:32.658997+0000","last_fullsized":"2023-12-16T21:15:32.658997+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:29:50.196142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657047+0000","last_change":"2023-12-16T21:15:32.644091+0000","last_active":"2023-12-16T21:15:32.657047+0000","last_peered":"2023-12-16T21:15:32.657047+0000","last_clean":"2023-12-16T21:15:32.657047+0000","last_became_active":"2023-12-16T21:15:32.643472+0000","last_became_peered":"2023-12-16T21:15:32.643472+0000","last_unstale":"2023-12-16T21:15:32.657047+0000","last_undegraded":"2023-12-16T21:15:32.657047+0000","last_fullsized":"2023-12-16T21:15:32.657047+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.652474+0000","last_change":"2023-12-16T21:15:31.631729+0000","last_active":"2023-12-16T21:15:32.652474+0000","last_peered":"2023-12-16T21:15:32.652474+0000","last_clean":"2023-12-16T21:15:32.652474+0000","last_became_active":"2023-12-16T21:15:31.631438+0000","last_became_peered":"2023-12-16T21:15:31.631438+0000","last_unstale":"2023-12-16T21:15:32.652474+0000","last_undegraded":"2023-12-16T21:15:32.652474+0000","last_fullsized":"2023-12-16T21:15:32.652474+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:01.579606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654948+0000","last_change":"2023-12-16T21:15:32.651492+0000","last_active":"2023-12-16T21:15:32.654948+0000","last_peered":"2023-12-16T21:15:32.654948+0000","last_clean":"2023-12-16T21:15:32.654948+0000","last_became_active":"2023-12-16T21:15:32.651209+0000","last_became_peered":"2023-12-16T21:15:32.651209+0000","last_unstale":"2023-12-16T21:15:32.654948+0000","last_undegraded":"2023-12-16T21:15:32.654948+0000","last_fullsized":"2023-12-16T21:15:32.654948+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:28:33.936466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656234+0000","last_change":"2023-12-16T21:15:31.635935+0000","last_active":"2023-12-16T21:15:32.656234+0000","last_peered":"2023-12-16T21:15:32.656234+0000","last_clean":"2023-12-16T21:15:32.656234+0000","last_became_active":"2023-12-16T21:15:31.635557+0000","last_became_peered":"2023-12-16T21:15:31.635557+0000","last_unstale":"2023-12-16T21:15:32.656234+0000","last_undegraded":"2023-12-16T21:15:32.656234+0000","last_fullsized":"2023-12-16T21:15:32.656234+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:07:18.410105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.653022+0000","last_change":"2023-12-16T21:15:32.644054+0000","last_active":"2023-12-16T21:15:32.653022+0000","last_peered":"2023-12-16T21:15:32.653022+0000","last_clean":"2023-12-16T21:15:32.653022+0000","last_became_active":"2023-12-16T21:15:32.643780+0000","last_became_peered":"2023-12-16T21:15:32.643780+0000","last_unstale":"2023-12-16T21:15:32.653022+0000","last_undegraded":"2023-12-16T21:15:32.653022+0000","last_fullsized":"2023-12-16T21:15:32.653022+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:54:35.100969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.640170+0000","last_change":"2023-12-16T21:15:31.637648+0000","last_active":"2023-12-16T21:15:32.640170+0000","last_peered":"2023-12-16T21:15:32.640170+0000","last_clean":"2023-12-16T21:15:32.640170+0000","last_became_active":"2023-12-16T21:15:31.637299+0000","last_became_peered":"2023-12-16T21:15:31.637299+0000","last_unstale":"2023-12-16T21:15:32.640170+0000","last_undegraded":"2023-12-16T21:15:32.640170+0000","last_fullsized":"2023-12-16T21:15:32.640170+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:26:43.428510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.652616+0000","last_change":"2023-12-16T21:15:32.650271+0000","last_active":"2023-12-16T21:15:32.652616+0000","last_peered":"2023-12-16T21:15:32.652616+0000","last_clean":"2023-12-16T21:15:32.652616+0000","last_became_active":"2023-12-16T21:15:32.650029+0000","last_became_peered":"2023-12-16T21:15:32.650029+0000","last_unstale":"2023-12-16T21:15:32.652616+0000","last_undegraded":"2023-12-16T21:15:32.652616+0000","last_fullsized":"2023-12-16T21:15:32.652616+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654240+0000","last_change":"2023-12-16T21:15:31.639259+0000","last_active":"2023-12-16T21:15:32.654240+0000","last_peered":"2023-12-16T21:15:32.654240+0000","last_clean":"2023-12-16T21:15:32.654240+0000","last_became_active":"2023-12-16T21:15:31.638964+0000","last_became_peered":"2023-12-16T21:15:31.638964+0000","last_unstale":"2023-12-16T21:15:32.654240+0000","last_undegraded":"2023-12-16T21:15:32.654240+0000","last_fullsized":"2023-12-16T21:15:32.654240+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:38.157109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.649127+0000","last_change":"2023-12-16T21:15:32.648189+0000","last_active":"2023-12-16T21:15:32.649127+0000","last_peered":"2023-12-16T21:15:32.649127+0000","last_clean":"2023-12-16T21:15:32.649127+0000","last_became_active":"2023-12-16T21:15:32.647661+0000","last_became_peered":"2023-12-16T21:15:32.647661+0000","last_unstale":"2023-12-16T21:15:32.649127+0000","last_undegraded":"2023-12-16T21:15:32.649127+0000","last_fullsized":"2023-12-16T21:15:32.649127+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:20:10.477992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.637218+0000","last_change":"2023-12-16T21:15:31.632812+0000","last_active":"2023-12-16T21:15:32.637218+0000","last_peered":"2023-12-16T21:15:32.637218+0000","last_clean":"2023-12-16T21:15:32.637218+0000","last_became_active":"2023-12-16T21:15:31.632110+0000","last_became_peered":"2023-12-16T21:15:31.632110+0000","last_unstale":"2023-12-16T21:15:32.637218+0000","last_undegraded":"2023-12-16T21:15:32.637218+0000","last_fullsized":"2023-12-16T21:15:32.637218+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:31:01.329665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656129+0000","last_change":"2023-12-16T21:15:32.648321+0000","last_active":"2023-12-16T21:15:32.656129+0000","last_peered":"2023-12-16T21:15:32.656129+0000","last_clean":"2023-12-16T21:15:32.656129+0000","last_became_active":"2023-12-16T21:15:32.648022+0000","last_became_peered":"2023-12-16T21:15:32.648022+0000","last_unstale":"2023-12-16T21:15:32.656129+0000","last_undegraded":"2023-12-16T21:15:32.656129+0000","last_fullsized":"2023-12-16T21:15:32.656129+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:35.587180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.630267+0000","last_change":"2023-12-16T21:15:31.642923+0000","last_active":"2023-12-16T21:15:32.630267+0000","last_peered":"2023-12-16T21:15:32.630267+0000","last_clean":"2023-12-16T21:15:32.630267+0000","last_became_active":"2023-12-16T21:15:31.642612+0000","last_became_peered":"2023-12-16T21:15:31.642612+0000","last_unstale":"2023-12-16T21:15:32.630267+0000","last_undegraded":"2023-12-16T21:15:32.630267+0000","last_fullsized":"2023-12-16T21:15:32.630267+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:14:00.555170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.655845+0000","last_change":"2023-12-16T21:15:32.648680+0000","last_active":"2023-12-16T21:15:32.655845+0000","last_peered":"2023-12-16T21:15:32.655845+0000","last_clean":"2023-12-16T21:15:32.655845+0000","last_became_active":"2023-12-16T21:15:32.648397+0000","last_became_peered":"2023-12-16T21:15:32.648397+0000","last_unstale":"2023-12-16T21:15:32.655845+0000","last_undegraded":"2023-12-16T21:15:32.655845+0000","last_fullsized":"2023-12-16T21:15:32.655845+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.633038+0000","last_change":"2023-12-16T21:15:31.633820+0000","last_active":"2023-12-16T21:15:32.633038+0000","last_peered":"2023-12-16T21:15:32.633038+0000","last_clean":"2023-12-16T21:15:32.633038+0000","last_became_active":"2023-12-16T21:15:31.633526+0000","last_became_peered":"2023-12-16T21:15:31.633526+0000","last_unstale":"2023-12-16T21:15:32.633038+0000","last_undegraded":"2023-12-16T21:15:32.633038+0000","last_fullsized":"2023-12-16T21:15:32.633038+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:59:19.041122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.651255+0000","last_change":"2023-12-16T21:15:32.646932+0000","last_active":"2023-12-16T21:15:32.651255+0000","last_peered":"2023-12-16T21:15:32.651255+0000","last_clean":"2023-12-16T21:15:32.651255+0000","last_became_active":"2023-12-16T21:15:32.646676+0000","last_became_peered":"2023-12-16T21:15:32.646676+0000","last_unstale":"2023-12-16T21:15:32.651255+0000","last_undegraded":"2023-12-16T21:15:32.651255+0000","last_fullsized":"2023-12-16T21:15:32.651255+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658055+0000","last_change":"2023-12-16T21:15:31.639980+0000","last_active":"2023-12-16T21:15:32.658055+0000","last_peered":"2023-12-16T21:15:32.658055+0000","last_clean":"2023-12-16T21:15:32.658055+0000","last_became_active":"2023-12-16T21:15:31.639651+0000","last_became_peered":"2023-12-16T21:15:31.639651+0000","last_unstale":"2023-12-16T21:15:32.658055+0000","last_undegraded":"2023-12-16T21:15:32.658055+0000","last_fullsized":"2023-12-16T21:15:32.658055+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:53:34.629855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648167+0000","last_change":"2023-12-16T21:15:32.643658+0000","last_active":"2023-12-16T21:15:32.648167+0000","last_peered":"2023-12-16T21:15:32.648167+0000","last_clean":"2023-12-16T21:15:32.648167+0000","last_became_active":"2023-12-16T21:15:32.643337+0000","last_became_peered":"2023-12-16T21:15:32.643337+0000","last_unstale":"2023-12-16T21:15:32.648167+0000","last_undegraded":"2023-12-16T21:15:32.648167+0000","last_fullsized":"2023-12-16T21:15:32.648167+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:35.587180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.636129+0000","last_change":"2023-12-16T21:15:31.634228+0000","last_active":"2023-12-16T21:15:32.636129+0000","last_peered":"2023-12-16T21:15:32.636129+0000","last_clean":"2023-12-16T21:15:32.636129+0000","last_became_active":"2023-12-16T21:15:31.633943+0000","last_became_peered":"2023-12-16T21:15:31.633943+0000","last_unstale":"2023-12-16T21:15:32.636129+0000","last_undegraded":"2023-12-16T21:15:32.636129+0000","last_fullsized":"2023-12-16T21:15:32.636129+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:20:51.013316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658551+0000","last_change":"2023-12-16T21:15:32.653418+0000","last_active":"2023-12-16T21:15:32.658551+0000","last_peered":"2023-12-16T21:15:32.658551+0000","last_clean":"2023-12-16T21:15:32.658551+0000","last_became_active":"2023-12-16T21:15:32.653146+0000","last_became_peered":"2023-12-16T21:15:32.653146+0000","last_unstale":"2023-12-16T21:15:32.658551+0000","last_undegraded":"2023-12-16T21:15:32.658551+0000","last_fullsized":"2023-12-16T21:15:32.658551+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:31:00.145990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.631812+0000","last_change":"2023-12-16T21:15:31.641639+0000","last_active":"2023-12-16T21:15:32.631812+0000","last_peered":"2023-12-16T21:15:32.631812+0000","last_clean":"2023-12-16T21:15:32.631812+0000","last_became_active":"2023-12-16T21:15:31.641118+0000","last_became_peered":"2023-12-16T21:15:31.641118+0000","last_unstale":"2023-12-16T21:15:32.631812+0000","last_undegraded":"2023-12-16T21:15:32.631812+0000","last_fullsized":"2023-12-16T21:15:32.631812+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:32:57.874803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657037+0000","last_change":"2023-12-16T21:15:32.649554+0000","last_active":"2023-12-16T21:15:32.657037+0000","last_peered":"2023-12-16T21:15:32.657037+0000","last_clean":"2023-12-16T21:15:32.657037+0000","last_became_active":"2023-12-16T21:15:32.649294+0000","last_became_peered":"2023-12-16T21:15:32.649294+0000","last_unstale":"2023-12-16T21:15:32.657037+0000","last_undegraded":"2023-12-16T21:15:32.657037+0000","last_fullsized":"2023-12-16T21:15:32.657037+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:48.000718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.638090+0000","last_change":"2023-12-16T21:15:31.633388+0000","last_active":"2023-12-16T21:15:32.638090+0000","last_peered":"2023-12-16T21:15:32.638090+0000","last_clean":"2023-12-16T21:15:32.638090+0000","last_became_active":"2023-12-16T21:15:31.633135+0000","last_became_peered":"2023-12-16T21:15:31.633135+0000","last_unstale":"2023-12-16T21:15:32.638090+0000","last_undegraded":"2023-12-16T21:15:32.638090+0000","last_fullsized":"2023-12-16T21:15:32.638090+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:25.748300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656448+0000","last_change":"2023-12-16T21:15:32.654941+0000","last_active":"2023-12-16T21:15:32.656448+0000","last_peered":"2023-12-16T21:15:32.656448+0000","last_clean":"2023-12-16T21:15:32.656448+0000","last_became_active":"2023-12-16T21:15:32.654663+0000","last_became_peered":"2023-12-16T21:15:32.654663+0000","last_unstale":"2023-12-16T21:15:32.656448+0000","last_undegraded":"2023-12-16T21:15:32.656448+0000","last_fullsized":"2023-12-16T21:15:32.656448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:43.155045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.637375+0000","last_change":"2023-12-16T21:15:31.641281+0000","last_active":"2023-12-16T21:15:32.637375+0000","last_peered":"2023-12-16T21:15:32.637375+0000","last_clean":"2023-12-16T21:15:32.637375+0000","last_became_active":"2023-12-16T21:15:31.640470+0000","last_became_peered":"2023-12-16T21:15:31.640470+0000","last_unstale":"2023-12-16T21:15:32.637375+0000","last_undegraded":"2023-12-16T21:15:32.637375+0000","last_fullsized":"2023-12-16T21:15:32.637375+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:37:44.577523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647653+0000","last_change":"2023-12-16T21:15:32.645684+0000","last_active":"2023-12-16T21:15:32.647653+0000","last_peered":"2023-12-16T21:15:32.647653+0000","last_clean":"2023-12-16T21:15:32.647653+0000","last_became_active":"2023-12-16T21:15:32.645394+0000","last_became_peered":"2023-12-16T21:15:32.645394+0000","last_unstale":"2023-12-16T21:15:32.647653+0000","last_undegraded":"2023-12-16T21:15:32.647653+0000","last_fullsized":"2023-12-16T21:15:32.647653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.633906+0000","last_change":"2023-12-16T21:15:31.640752+0000","last_active":"2023-12-16T21:15:32.633906+0000","last_peered":"2023-12-16T21:15:32.633906+0000","last_clean":"2023-12-16T21:15:32.633906+0000","last_became_active":"2023-12-16T21:15:31.640513+0000","last_became_peered":"2023-12-16T21:15:31.640513+0000","last_unstale":"2023-12-16T21:15:32.633906+0000","last_undegraded":"2023-12-16T21:15:32.633906+0000","last_fullsized":"2023-12-16T21:15:32.633906+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:25:42.211225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657885+0000","last_change":"2023-12-16T21:15:32.648992+0000","last_active":"2023-12-16T21:15:32.657885+0000","last_peered":"2023-12-16T21:15:32.657885+0000","last_clean":"2023-12-16T21:15:32.657885+0000","last_became_active":"2023-12-16T21:15:32.648693+0000","last_became_peered":"2023-12-16T21:15:32.648693+0000","last_unstale":"2023-12-16T21:15:32.657885+0000","last_undegraded":"2023-12-16T21:15:32.657885+0000","last_fullsized":"2023-12-16T21:15:32.657885+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:13:30.242051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.634605+0000","last_change":"2023-12-16T21:15:31.634447+0000","last_active":"2023-12-16T21:15:32.634605+0000","last_peered":"2023-12-16T21:15:32.634605+0000","last_clean":"2023-12-16T21:15:32.634605+0000","last_became_active":"2023-12-16T21:15:31.634139+0000","last_became_peered":"2023-12-16T21:15:31.634139+0000","last_unstale":"2023-12-16T21:15:32.634605+0000","last_undegraded":"2023-12-16T21:15:32.634605+0000","last_fullsized":"2023-12-16T21:15:32.634605+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:34.577140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.649174+0000","last_change":"2023-12-16T21:15:32.645298+0000","last_active":"2023-12-16T21:15:32.649174+0000","last_peered":"2023-12-16T21:15:32.649174+0000","last_clean":"2023-12-16T21:15:32.649174+0000","last_became_active":"2023-12-16T21:15:32.645031+0000","last_became_peered":"2023-12-16T21:15:32.645031+0000","last_unstale":"2023-12-16T21:15:32.649174+0000","last_undegraded":"2023-12-16T21:15:32.649174+0000","last_fullsized":"2023-12-16T21:15:32.649174+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:58:44.030796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.631656+0000","last_change":"2023-12-16T21:15:31.633038+0000","last_active":"2023-12-16T21:15:32.631656+0000","last_peered":"2023-12-16T21:15:32.631656+0000","last_clean":"2023-12-16T21:15:32.631656+0000","last_became_active":"2023-12-16T21:15:31.632736+0000","last_became_peered":"2023-12-16T21:15:31.632736+0000","last_unstale":"2023-12-16T21:15:32.631656+0000","last_undegraded":"2023-12-16T21:15:32.631656+0000","last_fullsized":"2023-12-16T21:15:32.631656+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:42.145005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":190,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-16T21:15:31.629355+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:15:31.629355+0000","last_peered":"2023-12-16T21:15:31.629355+0000","last_clean":"2023-12-16T21:15:31.629355+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:15:31.629355+0000","last_undegraded":"2023-12-16T21:15:31.629355+0000","last_fullsized":"2023-12-16T21:15:31.629355+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":830638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67800000000000005,"5min":1.159,"15min":1.159},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69299999999999995,"5min":1.181,"15min":1.181},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.624,"15min":0.624},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.749,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.503}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67700000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.627}]},{"osd":6,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.40300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56999999999999995}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48499999999999999,"15min":0.47199999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.755,"15min":0.755},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.505}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81399999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.627},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.42099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.625},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.498},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.70899999999999996,"15min":0.73899999999999999},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46700000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.39600000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.438},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.752,"15min":0.752},"last":0.46700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":585728,"data_stored":240852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.621}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.435}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.496},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46500000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54800000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.251},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.66100000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":593920,"data_stored":241882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68700000000000006,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.433,"15min":0.438},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58499999999999996,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.311},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":1.367,"15min":1.367},"last":0.34399999999999997}]},{"osd":3,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65700000000000003,"5min":1.46,"15min":1.46},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.874,"15min":0.874},"last":0.67300000000000004}]},{"osd":5,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.78000000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.62,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.76800000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":830694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.248}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.37}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46500000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.42299999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:15:35.643 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:15:35.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.786+0000 7f6c2a26b640 1 Processor -- start 2023-12-16T21:15:35.786 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.786+0000 7f6c2a26b640 1 -- start start 2023-12-16T21:15:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c2a26b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6c24146460 0x7f6c240b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c2a26b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 0x7f6c2406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c2a26b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24146fd0 con 0x7f6c2406eab0 2023-12-16T21:15:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c2a26b640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c2406c180 con 0x7f6c24146460 2023-12-16T21:15:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c2a26b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2406c460 con 0x7f6c24146840 2023-12-16T21:15:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c28808640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6c2406eab0 0x7f6c24160f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34458/0 (socket says 172.21.15.46:34458) 2023-12-16T21:15:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c28808640 1 -- 172.21.15.46:0/1773799674 learned_addr learned my addr 172.21.15.46:0/1773799674 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c237fe640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 0x7f6c2406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c23fff640 1 --2- 172.21.15.46:0/1773799674 >> v2:172.21.15.121:3300/0 conn(0x7f6c24146460 0x7f6c240b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.787+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785425604 0 0) 0x7f6c24146fd0 con 0x7f6c2406eab0 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c0c003520 con 0x7f6c2406eab0 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 232356239 0 0) 0x7f6c0c003520 con 0x7f6c2406eab0 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 msgr2=0x7f6c2406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 0x7f6c2406baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 >> v2:172.21.15.121:3300/0 conn(0x7f6c24146460 msgr2=0x7f6c240b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 --2- 172.21.15.46:0/1773799674 >> v2:172.21.15.121:3300/0 conn(0x7f6c24146460 0x7f6c240b49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2406c740 con 0x7f6c2406eab0 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f6c1c0019f0 con 0x7f6c2406eab0 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 >> v1:172.21.15.46:6789/0 conn(0x7f6c2406eab0 legacy=0x7f6c24160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 0x7f6c0c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c0c004d00 0x7f6c0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2406c180 con 0x7f6c0c003ef0 2023-12-16T21:15:35.791 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2406c460 con 0x7f6c0c004d00 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c00b980 con 0x7f6c24146840 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c237fe640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c23fff640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 0x7f6c0c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c28808640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c0c004d00 0x7f6c0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c237fe640 1 -- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 msgr2=0x7f6c0c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c237fe640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c24146840 0x7f6c0c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c237fe640 1 -- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c0c004d00 msgr2=0x7f6c0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c237fe640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c0c004d00 0x7f6c0c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.788+0000 7f6c237fe640 1 -- 172.21.15.46:0/1773799674 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c003520 con 0x7f6c0c003ef0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.789+0000 7f6c237fe640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c0c0047c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009a90 tx=0x7f6c0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6599ced3934db419 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.789+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1402c6b0 con 0x7f6c0c003ef0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.789+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1402c810 con 0x7f6c0c003ef0 2023-12-16T21:15:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.789+0000 7f6c22ffd640 1 -- 172.21.15.46:0/1773799674 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1402cbc0 con 0x7f6c0c003ef0 2023-12-16T21:15:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c2a26b640 1 -- 172.21.15.46:0/1773799674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 msgr2=0x7f6c0c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/1773799674 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c0c0047c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009a90 tx=0x7f6c0c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c28808640 1 -- 172.21.15.46:0/1773799674 reap_dead start 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c2a26b640 1 -- 172.21.15.46:0/1773799674 shutdown_connections 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c2a26b640 1 -- 172.21.15.46:0/1773799674 >> 172.21.15.46:0/1773799674 conn(0x7f6c2406a6b0 msgr2=0x7f6c2406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c2a26b640 1 -- 172.21.15.46:0/1773799674 shutdown_connections 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.790+0000 7f6c2a26b640 1 -- 172.21.15.46:0/1773799674 wait complete. 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.791+0000 7f6c2a26b640 1 Processor -- start 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.791+0000 7f6c2a26b640 1 -- start start 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c2a26b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c240b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c2a26b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c240b2880 0x7f6c241f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c2a26b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c241f3680 0x7f6c241326d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c2a26b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240b2d00 con 0x7f6c0c003ef0 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c2a26b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240cc460 con 0x7f6c241f3680 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c2a26b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240cb850 con 0x7f6c240b2880 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c240b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c240b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59750/0 (socket says 172.21.15.46:59750) 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 -- 172.21.15.46:0/3444006278 learned_addr learned my addr 172.21.15.46:0/3444006278 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 -- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c240b2880 msgr2=0x7f6c241f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c240b2880 0x7f6c241f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 -- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c241f3680 msgr2=0x7f6c241326d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c241f3680 0x7f6c241326d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 -- 172.21.15.46:0/3444006278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240cbb30 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.792+0000 7f6c23fff640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c240b2340 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6c18011a70 tx=0x7f6c18011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.793+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c18009680 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.793+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1800fad0 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.793+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c240cbe10 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.793+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1800fe60 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.793+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c240d1340 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.794+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24069400 con 0x7f6c0c003ef0 2023-12-16T21:15:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.794+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6c18014440 con 0x7f6c0c003ef0 2023-12-16T21:15:35.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.794+0000 7f6c20ff9640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6c1006ae70 0x7f6c1006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:35.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.795+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c18089e80 con 0x7f6c0c003ef0 2023-12-16T21:15:35.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.795+0000 7f6c237fe640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6c1006ae70 0x7f6c1006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:35.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.795+0000 7f6c237fe640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6c1006ae70 0x7f6c1006d330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6c14036000 tx=0x7f6c1402e000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:35.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.799+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6c18054f30 con 0x7f6c0c003ef0 2023-12-16T21:15:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.971+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c2406cd00 con 0x7f6c1006ae70 2023-12-16T21:15:35.978 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.979+0000 7f6c20ff9640 1 -- 172.21.15.46:0/3444006278 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289405 (secure 0 0 0) 0x7f6c14040450 con 0x7f6c1006ae70 2023-12-16T21:15:35.982 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:35.983 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:15:35.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6c1006ae70 msgr2=0x7f6c1006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6c1006ae70 0x7f6c1006d330 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6c14036000 tx=0x7f6c1402e000 comp rx=0 tx=0).stop 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 msgr2=0x7f6c240b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c240b2340 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6c18011a70 tx=0x7f6c18011f30 comp rx=0 tx=0).stop 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 shutdown_connections 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6c1006ae70 0x7f6c1006d330 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c241f3680 0x7f6c241326d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c240b2880 0x7f6c241f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 --2- 172.21.15.46:0/3444006278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c0c003ef0 0x7f6c240b2340 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 >> 172.21.15.46:0/3444006278 conn(0x7f6c2406a6b0 msgr2=0x7f6c240b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 shutdown_connections 2023-12-16T21:15:35.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:35.982+0000 7f6c2a26b640 1 -- 172.21.15.46:0/3444006278 wait complete. 2023-12-16T21:15:35.993 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2023-12-16T21:15:33.983433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":89,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238520,"kb_used_data":5680,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736200,"statfs":{"total":773094113280,"available":772849868800,"internally_reserved":0,"allocated":5816320,"data_stored":3078046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000831"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648563+0000","last_change":"2023-12-16T21:15:32.642665+0000","last_active":"2023-12-16T21:15:32.648563+0000","last_peered":"2023-12-16T21:15:32.648563+0000","last_clean":"2023-12-16T21:15:32.648563+0000","last_became_active":"2023-12-16T21:15:32.642356+0000","last_became_peered":"2023-12-16T21:15:32.642356+0000","last_unstale":"2023-12-16T21:15:32.648563+0000","last_undegraded":"2023-12-16T21:15:32.648563+0000","last_fullsized":"2023-12-16T21:15:32.648563+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:47:10.770226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654866+0000","last_change":"2023-12-16T21:15:32.651177+0000","last_active":"2023-12-16T21:15:32.654866+0000","last_peered":"2023-12-16T21:15:32.654866+0000","last_clean":"2023-12-16T21:15:32.654866+0000","last_became_active":"2023-12-16T21:15:32.650714+0000","last_became_peered":"2023-12-16T21:15:32.650714+0000","last_unstale":"2023-12-16T21:15:32.654866+0000","last_undegraded":"2023-12-16T21:15:32.654866+0000","last_fullsized":"2023-12-16T21:15:32.654866+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654479+0000","last_change":"2023-12-16T21:15:32.646414+0000","last_active":"2023-12-16T21:15:32.654479+0000","last_peered":"2023-12-16T21:15:32.654479+0000","last_clean":"2023-12-16T21:15:32.654479+0000","last_became_active":"2023-12-16T21:15:32.646132+0000","last_became_peered":"2023-12-16T21:15:32.646132+0000","last_unstale":"2023-12-16T21:15:32.654479+0000","last_undegraded":"2023-12-16T21:15:32.654479+0000","last_fullsized":"2023-12-16T21:15:32.654479+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:25:43.221265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654680+0000","last_change":"2023-12-16T21:15:32.650750+0000","last_active":"2023-12-16T21:15:32.654680+0000","last_peered":"2023-12-16T21:15:32.654680+0000","last_clean":"2023-12-16T21:15:32.654680+0000","last_became_active":"2023-12-16T21:15:32.650470+0000","last_became_peered":"2023-12-16T21:15:32.650470+0000","last_unstale":"2023-12-16T21:15:32.654680+0000","last_undegraded":"2023-12-16T21:15:32.654680+0000","last_fullsized":"2023-12-16T21:15:32.654680+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:35.587180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654925+0000","last_change":"2023-12-16T21:15:32.652762+0000","last_active":"2023-12-16T21:15:32.654925+0000","last_peered":"2023-12-16T21:15:32.654925+0000","last_clean":"2023-12-16T21:15:32.654925+0000","last_became_active":"2023-12-16T21:15:32.652505+0000","last_became_peered":"2023-12-16T21:15:32.652505+0000","last_unstale":"2023-12-16T21:15:32.654925+0000","last_undegraded":"2023-12-16T21:15:32.654925+0000","last_fullsized":"2023-12-16T21:15:32.654925+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.646166+0000","last_change":"2023-12-16T21:15:32.639650+0000","last_active":"2023-12-16T21:15:32.646166+0000","last_peered":"2023-12-16T21:15:32.646166+0000","last_clean":"2023-12-16T21:15:32.646166+0000","last_became_active":"2023-12-16T21:15:32.639354+0000","last_became_peered":"2023-12-16T21:15:32.639354+0000","last_unstale":"2023-12-16T21:15:32.646166+0000","last_undegraded":"2023-12-16T21:15:32.646166+0000","last_fullsized":"2023-12-16T21:15:32.646166+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647215+0000","last_change":"2023-12-16T21:15:32.644479+0000","last_active":"2023-12-16T21:15:32.647215+0000","last_peered":"2023-12-16T21:15:32.647215+0000","last_clean":"2023-12-16T21:15:32.647215+0000","last_became_active":"2023-12-16T21:15:32.644048+0000","last_became_peered":"2023-12-16T21:15:32.644048+0000","last_unstale":"2023-12-16T21:15:32.647215+0000","last_undegraded":"2023-12-16T21:15:32.647215+0000","last_fullsized":"2023-12-16T21:15:32.647215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:02:52.674851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659252+0000","last_change":"2023-12-16T21:15:32.651612+0000","last_active":"2023-12-16T21:15:32.659252+0000","last_peered":"2023-12-16T21:15:32.659252+0000","last_clean":"2023-12-16T21:15:32.659252+0000","last_became_active":"2023-12-16T21:15:32.651310+0000","last_became_peered":"2023-12-16T21:15:32.651310+0000","last_unstale":"2023-12-16T21:15:32.659252+0000","last_undegraded":"2023-12-16T21:15:32.659252+0000","last_fullsized":"2023-12-16T21:15:32.659252+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:31:52.341272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658218+0000","last_change":"2023-12-16T21:15:32.651032+0000","last_active":"2023-12-16T21:15:32.658218+0000","last_peered":"2023-12-16T21:15:32.658218+0000","last_clean":"2023-12-16T21:15:32.658218+0000","last_became_active":"2023-12-16T21:15:32.650633+0000","last_became_peered":"2023-12-16T21:15:32.650633+0000","last_unstale":"2023-12-16T21:15:32.658218+0000","last_undegraded":"2023-12-16T21:15:32.658218+0000","last_fullsized":"2023-12-16T21:15:32.658218+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:26:37.791962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.646131+0000","last_change":"2023-12-16T21:15:32.637193+0000","last_active":"2023-12-16T21:15:32.646131+0000","last_peered":"2023-12-16T21:15:32.646131+0000","last_clean":"2023-12-16T21:15:32.646131+0000","last_became_active":"2023-12-16T21:15:32.636914+0000","last_became_peered":"2023-12-16T21:15:32.636914+0000","last_unstale":"2023-12-16T21:15:32.646131+0000","last_undegraded":"2023-12-16T21:15:32.646131+0000","last_fullsized":"2023-12-16T21:15:32.646131+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:14:01.565210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644927+0000","last_change":"2023-12-16T21:15:32.641734+0000","last_active":"2023-12-16T21:15:32.644927+0000","last_peered":"2023-12-16T21:15:32.644927+0000","last_clean":"2023-12-16T21:15:32.644927+0000","last_became_active":"2023-12-16T21:15:32.641452+0000","last_became_peered":"2023-12-16T21:15:32.641452+0000","last_unstale":"2023-12-16T21:15:32.644927+0000","last_undegraded":"2023-12-16T21:15:32.644927+0000","last_fullsized":"2023-12-16T21:15:32.644927+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:36:59.166199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658968+0000","last_change":"2023-12-16T21:15:32.656392+0000","last_active":"2023-12-16T21:15:32.658968+0000","last_peered":"2023-12-16T21:15:32.658968+0000","last_clean":"2023-12-16T21:15:32.658968+0000","last_became_active":"2023-12-16T21:15:32.655344+0000","last_became_peered":"2023-12-16T21:15:32.655344+0000","last_unstale":"2023-12-16T21:15:32.658968+0000","last_undegraded":"2023-12-16T21:15:32.658968+0000","last_fullsized":"2023-12-16T21:15:32.658968+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:15:38.975571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644680+0000","last_change":"2023-12-16T21:15:32.637168+0000","last_active":"2023-12-16T21:15:32.644680+0000","last_peered":"2023-12-16T21:15:32.644680+0000","last_clean":"2023-12-16T21:15:32.644680+0000","last_became_active":"2023-12-16T21:15:32.636875+0000","last_became_peered":"2023-12-16T21:15:32.636875+0000","last_unstale":"2023-12-16T21:15:32.644680+0000","last_undegraded":"2023-12-16T21:15:32.644680+0000","last_fullsized":"2023-12-16T21:15:32.644680+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:16.640850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647341+0000","last_change":"2023-12-16T21:15:32.644616+0000","last_active":"2023-12-16T21:15:32.647341+0000","last_peered":"2023-12-16T21:15:32.647341+0000","last_clean":"2023-12-16T21:15:32.647341+0000","last_became_active":"2023-12-16T21:15:32.644324+0000","last_became_peered":"2023-12-16T21:15:32.644324+0000","last_unstale":"2023-12-16T21:15:32.647341+0000","last_undegraded":"2023-12-16T21:15:32.647341+0000","last_fullsized":"2023-12-16T21:15:32.647341+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:16.640850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654557+0000","last_change":"2023-12-16T21:15:32.650698+0000","last_active":"2023-12-16T21:15:32.654557+0000","last_peered":"2023-12-16T21:15:32.654557+0000","last_clean":"2023-12-16T21:15:32.654557+0000","last_became_active":"2023-12-16T21:15:32.650389+0000","last_became_peered":"2023-12-16T21:15:32.650389+0000","last_unstale":"2023-12-16T21:15:32.654557+0000","last_undegraded":"2023-12-16T21:15:32.654557+0000","last_fullsized":"2023-12-16T21:15:32.654557+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:58:58.752236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648198+0000","last_change":"2023-12-16T21:15:32.641683+0000","last_active":"2023-12-16T21:15:32.648198+0000","last_peered":"2023-12-16T21:15:32.648198+0000","last_clean":"2023-12-16T21:15:32.648198+0000","last_became_active":"2023-12-16T21:15:32.641389+0000","last_became_peered":"2023-12-16T21:15:32.641389+0000","last_unstale":"2023-12-16T21:15:32.648198+0000","last_undegraded":"2023-12-16T21:15:32.648198+0000","last_fullsized":"2023-12-16T21:15:32.648198+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647655+0000","last_change":"2023-12-16T21:15:32.642697+0000","last_active":"2023-12-16T21:15:32.647655+0000","last_peered":"2023-12-16T21:15:32.647655+0000","last_clean":"2023-12-16T21:15:32.647655+0000","last_became_active":"2023-12-16T21:15:32.642400+0000","last_became_peered":"2023-12-16T21:15:32.642400+0000","last_unstale":"2023-12-16T21:15:32.647655+0000","last_undegraded":"2023-12-16T21:15:32.647655+0000","last_fullsized":"2023-12-16T21:15:32.647655+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:39.167149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659671+0000","last_change":"2023-12-16T21:15:32.653851+0000","last_active":"2023-12-16T21:15:32.659671+0000","last_peered":"2023-12-16T21:15:32.659671+0000","last_clean":"2023-12-16T21:15:32.659671+0000","last_became_active":"2023-12-16T21:15:32.653538+0000","last_became_peered":"2023-12-16T21:15:32.653538+0000","last_unstale":"2023-12-16T21:15:32.659671+0000","last_undegraded":"2023-12-16T21:15:32.659671+0000","last_fullsized":"2023-12-16T21:15:32.659671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:32:58.884843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.646552+0000","last_change":"2023-12-16T21:15:32.642522+0000","last_active":"2023-12-16T21:15:32.646552+0000","last_peered":"2023-12-16T21:15:32.646552+0000","last_clean":"2023-12-16T21:15:32.646552+0000","last_became_active":"2023-12-16T21:15:32.642213+0000","last_became_peered":"2023-12-16T21:15:32.642213+0000","last_unstale":"2023-12-16T21:15:32.646552+0000","last_undegraded":"2023-12-16T21:15:32.646552+0000","last_fullsized":"2023-12-16T21:15:32.646552+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:47:10.770226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657418+0000","last_change":"2023-12-16T21:15:32.652127+0000","last_active":"2023-12-16T21:15:32.657418+0000","last_peered":"2023-12-16T21:15:32.657418+0000","last_clean":"2023-12-16T21:15:32.657418+0000","last_became_active":"2023-12-16T21:15:32.651856+0000","last_became_peered":"2023-12-16T21:15:32.651856+0000","last_unstale":"2023-12-16T21:15:32.657418+0000","last_undegraded":"2023-12-16T21:15:32.657418+0000","last_fullsized":"2023-12-16T21:15:32.657418+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:25:44.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657363+0000","last_change":"2023-12-16T21:15:32.652066+0000","last_active":"2023-12-16T21:15:32.657363+0000","last_peered":"2023-12-16T21:15:32.657363+0000","last_clean":"2023-12-16T21:15:32.657363+0000","last_became_active":"2023-12-16T21:15:32.651792+0000","last_became_peered":"2023-12-16T21:15:32.651792+0000","last_unstale":"2023-12-16T21:15:32.657363+0000","last_undegraded":"2023-12-16T21:15:32.657363+0000","last_fullsized":"2023-12-16T21:15:32.657363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:20:39.668395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658325+0000","last_change":"2023-12-16T21:15:32.651318+0000","last_active":"2023-12-16T21:15:32.658325+0000","last_peered":"2023-12-16T21:15:32.658325+0000","last_clean":"2023-12-16T21:15:32.658325+0000","last_became_active":"2023-12-16T21:15:32.651011+0000","last_became_peered":"2023-12-16T21:15:32.651011+0000","last_unstale":"2023-12-16T21:15:32.658325+0000","last_undegraded":"2023-12-16T21:15:32.658325+0000","last_fullsized":"2023-12-16T21:15:32.658325+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:36:43.705381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644995+0000","last_change":"2023-12-16T21:15:32.641932+0000","last_active":"2023-12-16T21:15:32.644995+0000","last_peered":"2023-12-16T21:15:32.644995+0000","last_clean":"2023-12-16T21:15:32.644995+0000","last_became_active":"2023-12-16T21:15:32.641542+0000","last_became_peered":"2023-12-16T21:15:32.641542+0000","last_unstale":"2023-12-16T21:15:32.644995+0000","last_undegraded":"2023-12-16T21:15:32.644995+0000","last_fullsized":"2023-12-16T21:15:32.644995+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:16:33.086634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.637974+0000","last_change":"2023-12-16T21:15:32.637346+0000","last_active":"2023-12-16T21:15:32.637974+0000","last_peered":"2023-12-16T21:15:32.637974+0000","last_clean":"2023-12-16T21:15:32.637974+0000","last_became_active":"2023-12-16T21:15:32.637073+0000","last_became_peered":"2023-12-16T21:15:32.637073+0000","last_unstale":"2023-12-16T21:15:32.637974+0000","last_undegraded":"2023-12-16T21:15:32.637974+0000","last_fullsized":"2023-12-16T21:15:32.637974+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:17:27.143791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656159+0000","last_change":"2023-12-16T21:15:32.648955+0000","last_active":"2023-12-16T21:15:32.656159+0000","last_peered":"2023-12-16T21:15:32.656159+0000","last_clean":"2023-12-16T21:15:32.656159+0000","last_became_active":"2023-12-16T21:15:32.647872+0000","last_became_peered":"2023-12-16T21:15:32.647872+0000","last_unstale":"2023-12-16T21:15:32.656159+0000","last_undegraded":"2023-12-16T21:15:32.656159+0000","last_fullsized":"2023-12-16T21:15:32.656159+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657928+0000","last_change":"2023-12-16T21:15:32.650294+0000","last_active":"2023-12-16T21:15:32.657928+0000","last_peered":"2023-12-16T21:15:32.657928+0000","last_clean":"2023-12-16T21:15:32.657928+0000","last_became_active":"2023-12-16T21:15:32.650029+0000","last_became_peered":"2023-12-16T21:15:32.650029+0000","last_unstale":"2023-12-16T21:15:32.657928+0000","last_undegraded":"2023-12-16T21:15:32.657928+0000","last_fullsized":"2023-12-16T21:15:32.657928+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:51:15.168522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648880+0000","last_change":"2023-12-16T21:15:32.642557+0000","last_active":"2023-12-16T21:15:32.648880+0000","last_peered":"2023-12-16T21:15:32.648880+0000","last_clean":"2023-12-16T21:15:32.648880+0000","last_became_active":"2023-12-16T21:15:32.642284+0000","last_became_peered":"2023-12-16T21:15:32.642284+0000","last_unstale":"2023-12-16T21:15:32.648880+0000","last_undegraded":"2023-12-16T21:15:32.648880+0000","last_fullsized":"2023-12-16T21:15:32.648880+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:43.155045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.649080+0000","last_change":"2023-12-16T21:15:32.646191+0000","last_active":"2023-12-16T21:15:32.649080+0000","last_peered":"2023-12-16T21:15:32.649080+0000","last_clean":"2023-12-16T21:15:32.649080+0000","last_became_active":"2023-12-16T21:15:32.645731+0000","last_became_peered":"2023-12-16T21:15:32.645731+0000","last_unstale":"2023-12-16T21:15:32.649080+0000","last_undegraded":"2023-12-16T21:15:32.649080+0000","last_fullsized":"2023-12-16T21:15:32.649080+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:41:31.484631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656191+0000","last_change":"2023-12-16T21:15:32.648907+0000","last_active":"2023-12-16T21:15:32.656191+0000","last_peered":"2023-12-16T21:15:32.656191+0000","last_clean":"2023-12-16T21:15:32.656191+0000","last_became_active":"2023-12-16T21:15:32.648614+0000","last_became_peered":"2023-12-16T21:15:32.648614+0000","last_unstale":"2023-12-16T21:15:32.656191+0000","last_undegraded":"2023-12-16T21:15:32.656191+0000","last_fullsized":"2023-12-16T21:15:32.656191+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:54:35.100969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.650716+0000","last_change":"2023-12-16T21:15:32.646961+0000","last_active":"2023-12-16T21:15:32.650716+0000","last_peered":"2023-12-16T21:15:32.650716+0000","last_clean":"2023-12-16T21:15:32.650716+0000","last_became_active":"2023-12-16T21:15:32.646680+0000","last_became_peered":"2023-12-16T21:15:32.646680+0000","last_unstale":"2023-12-16T21:15:32.650716+0000","last_undegraded":"2023-12-16T21:15:32.650716+0000","last_fullsized":"2023-12-16T21:15:32.650716+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:43.155045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647671+0000","last_change":"2023-12-16T21:15:32.645008+0000","last_active":"2023-12-16T21:15:32.647671+0000","last_peered":"2023-12-16T21:15:32.647671+0000","last_clean":"2023-12-16T21:15:32.647671+0000","last_became_active":"2023-12-16T21:15:32.643968+0000","last_became_peered":"2023-12-16T21:15:32.643968+0000","last_unstale":"2023-12-16T21:15:32.647671+0000","last_undegraded":"2023-12-16T21:15:32.647671+0000","last_fullsized":"2023-12-16T21:15:32.647671+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:28:33.936466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.651276+0000","last_change":"2023-12-16T21:15:32.647529+0000","last_active":"2023-12-16T21:15:32.651276+0000","last_peered":"2023-12-16T21:15:32.651276+0000","last_clean":"2023-12-16T21:15:32.651276+0000","last_became_active":"2023-12-16T21:15:32.647260+0000","last_became_peered":"2023-12-16T21:15:32.647260+0000","last_unstale":"2023-12-16T21:15:32.651276+0000","last_undegraded":"2023-12-16T21:15:32.651276+0000","last_fullsized":"2023-12-16T21:15:32.651276+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659483+0000","last_change":"2023-12-16T21:15:32.652204+0000","last_active":"2023-12-16T21:15:32.659483+0000","last_peered":"2023-12-16T21:15:32.659483+0000","last_clean":"2023-12-16T21:15:32.659483+0000","last_became_active":"2023-12-16T21:15:32.651925+0000","last_became_peered":"2023-12-16T21:15:32.651925+0000","last_unstale":"2023-12-16T21:15:32.659483+0000","last_undegraded":"2023-12-16T21:15:32.659483+0000","last_fullsized":"2023-12-16T21:15:32.659483+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":15,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657510+0000","last_change":"2023-12-16T21:15:31.634592+0000","last_active":"2023-12-16T21:15:32.657510+0000","last_peered":"2023-12-16T21:15:32.657510+0000","last_clean":"2023-12-16T21:15:32.657510+0000","last_became_active":"2023-12-16T21:15:31.634363+0000","last_became_peered":"2023-12-16T21:15:31.634363+0000","last_unstale":"2023-12-16T21:15:32.657510+0000","last_undegraded":"2023-12-16T21:15:32.657510+0000","last_fullsized":"2023-12-16T21:15:32.657510+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:02:46.114548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644340+0000","last_change":"2023-12-16T21:15:32.641959+0000","last_active":"2023-12-16T21:15:32.644340+0000","last_peered":"2023-12-16T21:15:32.644340+0000","last_clean":"2023-12-16T21:15:32.644340+0000","last_became_active":"2023-12-16T21:15:32.641396+0000","last_became_peered":"2023-12-16T21:15:32.641396+0000","last_unstale":"2023-12-16T21:15:32.644340+0000","last_undegraded":"2023-12-16T21:15:32.644340+0000","last_fullsized":"2023-12-16T21:15:32.644340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658733+0000","last_change":"2023-12-16T21:15:31.643125+0000","last_active":"2023-12-16T21:15:32.658733+0000","last_peered":"2023-12-16T21:15:32.658733+0000","last_clean":"2023-12-16T21:15:32.658733+0000","last_became_active":"2023-12-16T21:15:31.642850+0000","last_became_peered":"2023-12-16T21:15:31.642850+0000","last_unstale":"2023-12-16T21:15:32.658733+0000","last_undegraded":"2023-12-16T21:15:32.658733+0000","last_fullsized":"2023-12-16T21:15:32.658733+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:25:42.211225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657492+0000","last_change":"2023-12-16T21:15:32.655542+0000","last_active":"2023-12-16T21:15:32.657492+0000","last_peered":"2023-12-16T21:15:32.657492+0000","last_clean":"2023-12-16T21:15:32.657492+0000","last_became_active":"2023-12-16T21:15:32.655222+0000","last_became_peered":"2023-12-16T21:15:32.655222+0000","last_unstale":"2023-12-16T21:15:32.657492+0000","last_undegraded":"2023-12-16T21:15:32.657492+0000","last_fullsized":"2023-12-16T21:15:32.657492+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:18:38.476202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657500+0000","last_change":"2023-12-16T21:15:31.636839+0000","last_active":"2023-12-16T21:15:32.657500+0000","last_peered":"2023-12-16T21:15:32.657500+0000","last_clean":"2023-12-16T21:15:32.657500+0000","last_became_active":"2023-12-16T21:15:31.635892+0000","last_became_peered":"2023-12-16T21:15:31.635892+0000","last_unstale":"2023-12-16T21:15:32.657500+0000","last_undegraded":"2023-12-16T21:15:32.657500+0000","last_fullsized":"2023-12-16T21:15:32.657500+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:46.990678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657820+0000","last_change":"2023-12-16T21:15:32.648190+0000","last_active":"2023-12-16T21:15:32.657820+0000","last_peered":"2023-12-16T21:15:32.657820+0000","last_clean":"2023-12-16T21:15:32.657820+0000","last_became_active":"2023-12-16T21:15:32.647878+0000","last_became_peered":"2023-12-16T21:15:32.647878+0000","last_unstale":"2023-12-16T21:15:32.657820+0000","last_undegraded":"2023-12-16T21:15:32.657820+0000","last_fullsized":"2023-12-16T21:15:32.657820+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:06:32.894407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659261+0000","last_change":"2023-12-16T21:15:31.637718+0000","last_active":"2023-12-16T21:15:32.659261+0000","last_peered":"2023-12-16T21:15:32.659261+0000","last_clean":"2023-12-16T21:15:32.659261+0000","last_became_active":"2023-12-16T21:15:31.637445+0000","last_became_peered":"2023-12-16T21:15:31.637445+0000","last_unstale":"2023-12-16T21:15:32.659261+0000","last_undegraded":"2023-12-16T21:15:32.659261+0000","last_fullsized":"2023-12-16T21:15:32.659261+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:32:57.874803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654260+0000","last_change":"2023-12-16T21:15:32.647424+0000","last_active":"2023-12-16T21:15:32.654260+0000","last_peered":"2023-12-16T21:15:32.654260+0000","last_clean":"2023-12-16T21:15:32.654260+0000","last_became_active":"2023-12-16T21:15:32.647149+0000","last_became_peered":"2023-12-16T21:15:32.647149+0000","last_unstale":"2023-12-16T21:15:32.654260+0000","last_undegraded":"2023-12-16T21:15:32.654260+0000","last_fullsized":"2023-12-16T21:15:32.654260+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659393+0000","last_change":"2023-12-16T21:15:31.638127+0000","last_active":"2023-12-16T21:15:32.659393+0000","last_peered":"2023-12-16T21:15:32.659393+0000","last_clean":"2023-12-16T21:15:32.659393+0000","last_became_active":"2023-12-16T21:15:31.637844+0000","last_became_peered":"2023-12-16T21:15:31.637844+0000","last_unstale":"2023-12-16T21:15:32.659393+0000","last_undegraded":"2023-12-16T21:15:32.659393+0000","last_fullsized":"2023-12-16T21:15:32.659393+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:20:51.013316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.651641+0000","last_change":"2023-12-16T21:15:32.648186+0000","last_active":"2023-12-16T21:15:32.651641+0000","last_peered":"2023-12-16T21:15:32.651641+0000","last_clean":"2023-12-16T21:15:32.651641+0000","last_became_active":"2023-12-16T21:15:32.647924+0000","last_became_peered":"2023-12-16T21:15:32.647924+0000","last_unstale":"2023-12-16T21:15:32.651641+0000","last_undegraded":"2023-12-16T21:15:32.651641+0000","last_fullsized":"2023-12-16T21:15:32.651641+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:48.000718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.650155+0000","last_change":"2023-12-16T21:15:31.633365+0000","last_active":"2023-12-16T21:15:32.650155+0000","last_peered":"2023-12-16T21:15:32.650155+0000","last_clean":"2023-12-16T21:15:32.650155+0000","last_became_active":"2023-12-16T21:15:31.633097+0000","last_became_peered":"2023-12-16T21:15:31.633097+0000","last_unstale":"2023-12-16T21:15:32.650155+0000","last_undegraded":"2023-12-16T21:15:32.650155+0000","last_fullsized":"2023-12-16T21:15:32.650155+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:14:00.555170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657867+0000","last_change":"2023-12-16T21:15:32.648913+0000","last_active":"2023-12-16T21:15:32.657867+0000","last_peered":"2023-12-16T21:15:32.657867+0000","last_clean":"2023-12-16T21:15:32.657867+0000","last_became_active":"2023-12-16T21:15:32.648611+0000","last_became_peered":"2023-12-16T21:15:32.648611+0000","last_unstale":"2023-12-16T21:15:32.657867+0000","last_undegraded":"2023-12-16T21:15:32.657867+0000","last_fullsized":"2023-12-16T21:15:32.657867+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:38:12.849334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.640878+0000","last_change":"2023-12-16T21:15:31.635059+0000","last_active":"2023-12-16T21:15:32.640878+0000","last_peered":"2023-12-16T21:15:32.640878+0000","last_clean":"2023-12-16T21:15:32.640878+0000","last_became_active":"2023-12-16T21:15:31.634198+0000","last_became_peered":"2023-12-16T21:15:31.634198+0000","last_unstale":"2023-12-16T21:15:32.640878+0000","last_undegraded":"2023-12-16T21:15:32.640878+0000","last_fullsized":"2023-12-16T21:15:32.640878+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:21:23.853508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654687+0000","last_change":"2023-12-16T21:15:32.651533+0000","last_active":"2023-12-16T21:15:32.654687+0000","last_peered":"2023-12-16T21:15:32.654687+0000","last_clean":"2023-12-16T21:15:32.654687+0000","last_became_active":"2023-12-16T21:15:32.651119+0000","last_became_peered":"2023-12-16T21:15:32.651119+0000","last_unstale":"2023-12-16T21:15:32.654687+0000","last_undegraded":"2023-12-16T21:15:32.654687+0000","last_fullsized":"2023-12-16T21:15:32.654687+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:29:51.206182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.629648+0000","last_change":"2023-12-16T21:15:31.634129+0000","last_active":"2023-12-16T21:15:32.629648+0000","last_peered":"2023-12-16T21:15:32.629648+0000","last_clean":"2023-12-16T21:15:32.629648+0000","last_became_active":"2023-12-16T21:15:31.633849+0000","last_became_peered":"2023-12-16T21:15:31.633849+0000","last_unstale":"2023-12-16T21:15:32.629648+0000","last_undegraded":"2023-12-16T21:15:32.629648+0000","last_fullsized":"2023-12-16T21:15:32.629648+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:22:43.310583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654215+0000","last_change":"2023-12-16T21:15:32.645960+0000","last_active":"2023-12-16T21:15:32.654215+0000","last_peered":"2023-12-16T21:15:32.654215+0000","last_clean":"2023-12-16T21:15:32.654215+0000","last_became_active":"2023-12-16T21:15:32.645705+0000","last_became_peered":"2023-12-16T21:15:32.645705+0000","last_unstale":"2023-12-16T21:15:32.654215+0000","last_undegraded":"2023-12-16T21:15:32.654215+0000","last_fullsized":"2023-12-16T21:15:32.654215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:54:35.100969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.636327+0000","last_change":"2023-12-16T21:15:31.636895+0000","last_active":"2023-12-16T21:15:32.636327+0000","last_peered":"2023-12-16T21:15:32.636327+0000","last_clean":"2023-12-16T21:15:32.636327+0000","last_became_active":"2023-12-16T21:15:31.636634+0000","last_became_peered":"2023-12-16T21:15:31.636634+0000","last_unstale":"2023-12-16T21:15:32.636327+0000","last_undegraded":"2023-12-16T21:15:32.636327+0000","last_fullsized":"2023-12-16T21:15:32.636327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:25:29.874049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.652590+0000","last_change":"2023-12-16T21:15:32.650407+0000","last_active":"2023-12-16T21:15:32.652590+0000","last_peered":"2023-12-16T21:15:32.652590+0000","last_clean":"2023-12-16T21:15:32.652590+0000","last_became_active":"2023-12-16T21:15:32.649946+0000","last_became_peered":"2023-12-16T21:15:32.649946+0000","last_unstale":"2023-12-16T21:15:32.652590+0000","last_undegraded":"2023-12-16T21:15:32.652590+0000","last_fullsized":"2023-12-16T21:15:32.652590+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:47:10.770226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.633754+0000","last_change":"2023-12-16T21:15:31.639976+0000","last_active":"2023-12-16T21:15:32.633754+0000","last_peered":"2023-12-16T21:15:32.633754+0000","last_clean":"2023-12-16T21:15:32.633754+0000","last_became_active":"2023-12-16T21:15:31.639673+0000","last_became_peered":"2023-12-16T21:15:31.639673+0000","last_unstale":"2023-12-16T21:15:32.633754+0000","last_undegraded":"2023-12-16T21:15:32.633754+0000","last_fullsized":"2023-12-16T21:15:32.633754+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:38.157109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657013+0000","last_change":"2023-12-16T21:15:32.648595+0000","last_active":"2023-12-16T21:15:32.657013+0000","last_peered":"2023-12-16T21:15:32.657013+0000","last_clean":"2023-12-16T21:15:32.657013+0000","last_became_active":"2023-12-16T21:15:32.648311+0000","last_became_peered":"2023-12-16T21:15:32.648311+0000","last_unstale":"2023-12-16T21:15:32.657013+0000","last_undegraded":"2023-12-16T21:15:32.657013+0000","last_fullsized":"2023-12-16T21:15:32.657013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":14,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656327+0000","last_change":"2023-12-16T21:15:31.641593+0000","last_active":"2023-12-16T21:15:32.656327+0000","last_peered":"2023-12-16T21:15:32.656327+0000","last_clean":"2023-12-16T21:15:32.656327+0000","last_became_active":"2023-12-16T21:15:31.641248+0000","last_became_peered":"2023-12-16T21:15:31.641248+0000","last_unstale":"2023-12-16T21:15:32.656327+0000","last_undegraded":"2023-12-16T21:15:32.656327+0000","last_fullsized":"2023-12-16T21:15:32.656327+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:40:03.528345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654301+0000","last_change":"2023-12-16T21:15:32.646803+0000","last_active":"2023-12-16T21:15:32.654301+0000","last_peered":"2023-12-16T21:15:32.654301+0000","last_clean":"2023-12-16T21:15:32.654301+0000","last_became_active":"2023-12-16T21:15:32.646493+0000","last_became_peered":"2023-12-16T21:15:32.646493+0000","last_unstale":"2023-12-16T21:15:32.654301+0000","last_undegraded":"2023-12-16T21:15:32.654301+0000","last_fullsized":"2023-12-16T21:15:32.654301+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:37:16.640850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":17,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.660889+0000","last_change":"2023-12-16T21:15:31.625494+0000","last_active":"2023-12-16T21:15:32.660889+0000","last_peered":"2023-12-16T21:15:32.660889+0000","last_clean":"2023-12-16T21:15:32.660889+0000","last_became_active":"2023-12-16T21:15:31.625179+0000","last_became_peered":"2023-12-16T21:15:31.625179+0000","last_unstale":"2023-12-16T21:15:32.660889+0000","last_undegraded":"2023-12-16T21:15:32.660889+0000","last_fullsized":"2023-12-16T21:15:32.660889+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:49:44.106454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657446+0000","last_change":"2023-12-16T21:15:32.655931+0000","last_active":"2023-12-16T21:15:32.657446+0000","last_peered":"2023-12-16T21:15:32.657446+0000","last_clean":"2023-12-16T21:15:32.657446+0000","last_became_active":"2023-12-16T21:15:32.655628+0000","last_became_peered":"2023-12-16T21:15:32.655628+0000","last_unstale":"2023-12-16T21:15:32.657446+0000","last_undegraded":"2023-12-16T21:15:32.657446+0000","last_fullsized":"2023-12-16T21:15:32.657446+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:48.000718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.660240+0000","last_change":"2023-12-16T21:15:31.633030+0000","last_active":"2023-12-16T21:15:32.660240+0000","last_peered":"2023-12-16T21:15:32.660240+0000","last_clean":"2023-12-16T21:15:32.660240+0000","last_became_active":"2023-12-16T21:15:31.632728+0000","last_became_peered":"2023-12-16T21:15:31.632728+0000","last_unstale":"2023-12-16T21:15:32.660240+0000","last_undegraded":"2023-12-16T21:15:32.660240+0000","last_fullsized":"2023-12-16T21:15:32.660240+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T05:40:03.528345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.644735+0000","last_change":"2023-12-16T21:15:32.640797+0000","last_active":"2023-12-16T21:15:32.644735+0000","last_peered":"2023-12-16T21:15:32.644735+0000","last_clean":"2023-12-16T21:15:32.644735+0000","last_became_active":"2023-12-16T21:15:32.640523+0000","last_became_peered":"2023-12-16T21:15:32.640523+0000","last_unstale":"2023-12-16T21:15:32.644735+0000","last_undegraded":"2023-12-16T21:15:32.644735+0000","last_fullsized":"2023-12-16T21:15:32.644735+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:58:58.752236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.640401+0000","last_change":"2023-12-16T21:15:31.635005+0000","last_active":"2023-12-16T21:15:32.640401+0000","last_peered":"2023-12-16T21:15:32.640401+0000","last_clean":"2023-12-16T21:15:32.640401+0000","last_became_active":"2023-12-16T21:15:31.634153+0000","last_became_peered":"2023-12-16T21:15:31.634153+0000","last_unstale":"2023-12-16T21:15:32.640401+0000","last_undegraded":"2023-12-16T21:15:32.640401+0000","last_fullsized":"2023-12-16T21:15:32.640401+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:44:27.335742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658477+0000","last_change":"2023-12-16T21:15:32.652820+0000","last_active":"2023-12-16T21:15:32.658477+0000","last_peered":"2023-12-16T21:15:32.658477+0000","last_clean":"2023-12-16T21:15:32.658477+0000","last_became_active":"2023-12-16T21:15:32.652314+0000","last_became_peered":"2023-12-16T21:15:32.652314+0000","last_unstale":"2023-12-16T21:15:32.658477+0000","last_undegraded":"2023-12-16T21:15:32.658477+0000","last_fullsized":"2023-12-16T21:15:32.658477+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:53:53.952000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.636461+0000","last_change":"2023-12-16T21:15:31.636870+0000","last_active":"2023-12-16T21:15:32.636461+0000","last_peered":"2023-12-16T21:15:32.636461+0000","last_clean":"2023-12-16T21:15:32.636461+0000","last_became_active":"2023-12-16T21:15:31.636576+0000","last_became_peered":"2023-12-16T21:15:31.636576+0000","last_unstale":"2023-12-16T21:15:32.636461+0000","last_undegraded":"2023-12-16T21:15:32.636461+0000","last_fullsized":"2023-12-16T21:15:32.636461+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:04.601140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647633+0000","last_change":"2023-12-16T21:15:32.642005+0000","last_active":"2023-12-16T21:15:32.647633+0000","last_peered":"2023-12-16T21:15:32.647633+0000","last_clean":"2023-12-16T21:15:32.647633+0000","last_became_active":"2023-12-16T21:15:32.641745+0000","last_became_peered":"2023-12-16T21:15:32.641745+0000","last_unstale":"2023-12-16T21:15:32.647633+0000","last_undegraded":"2023-12-16T21:15:32.647633+0000","last_fullsized":"2023-12-16T21:15:32.647633+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":19,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659176+0000","last_change":"2023-12-16T21:15:31.631537+0000","last_active":"2023-12-16T21:15:32.659176+0000","last_peered":"2023-12-16T21:15:32.659176+0000","last_clean":"2023-12-16T21:15:32.659176+0000","last_became_active":"2023-12-16T21:15:31.631249+0000","last_became_peered":"2023-12-16T21:15:31.631249+0000","last_unstale":"2023-12-16T21:15:32.659176+0000","last_undegraded":"2023-12-16T21:15:32.659176+0000","last_fullsized":"2023-12-16T21:15:32.659176+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:32:30.221788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.659265+0000","last_change":"2023-12-16T21:15:32.653334+0000","last_active":"2023-12-16T21:15:32.659265+0000","last_peered":"2023-12-16T21:15:32.659265+0000","last_clean":"2023-12-16T21:15:32.659265+0000","last_became_active":"2023-12-16T21:15:32.653088+0000","last_became_peered":"2023-12-16T21:15:32.653088+0000","last_unstale":"2023-12-16T21:15:32.659265+0000","last_undegraded":"2023-12-16T21:15:32.659265+0000","last_fullsized":"2023-12-16T21:15:32.659265+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:02:05.611180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658997+0000","last_change":"2023-12-16T21:15:31.638956+0000","last_active":"2023-12-16T21:15:32.658997+0000","last_peered":"2023-12-16T21:15:32.658997+0000","last_clean":"2023-12-16T21:15:32.658997+0000","last_became_active":"2023-12-16T21:15:31.638657+0000","last_became_peered":"2023-12-16T21:15:31.638657+0000","last_unstale":"2023-12-16T21:15:32.658997+0000","last_undegraded":"2023-12-16T21:15:32.658997+0000","last_fullsized":"2023-12-16T21:15:32.658997+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:29:50.196142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657047+0000","last_change":"2023-12-16T21:15:32.644091+0000","last_active":"2023-12-16T21:15:32.657047+0000","last_peered":"2023-12-16T21:15:32.657047+0000","last_clean":"2023-12-16T21:15:32.657047+0000","last_became_active":"2023-12-16T21:15:32.643472+0000","last_became_peered":"2023-12-16T21:15:32.643472+0000","last_unstale":"2023-12-16T21:15:32.657047+0000","last_undegraded":"2023-12-16T21:15:32.657047+0000","last_fullsized":"2023-12-16T21:15:32.657047+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:23:34.006954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.652474+0000","last_change":"2023-12-16T21:15:31.631729+0000","last_active":"2023-12-16T21:15:32.652474+0000","last_peered":"2023-12-16T21:15:32.652474+0000","last_clean":"2023-12-16T21:15:32.652474+0000","last_became_active":"2023-12-16T21:15:31.631438+0000","last_became_peered":"2023-12-16T21:15:31.631438+0000","last_unstale":"2023-12-16T21:15:32.652474+0000","last_undegraded":"2023-12-16T21:15:32.652474+0000","last_fullsized":"2023-12-16T21:15:32.652474+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T07:24:01.579606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654948+0000","last_change":"2023-12-16T21:15:32.651492+0000","last_active":"2023-12-16T21:15:32.654948+0000","last_peered":"2023-12-16T21:15:32.654948+0000","last_clean":"2023-12-16T21:15:32.654948+0000","last_became_active":"2023-12-16T21:15:32.651209+0000","last_became_peered":"2023-12-16T21:15:32.651209+0000","last_unstale":"2023-12-16T21:15:32.654948+0000","last_undegraded":"2023-12-16T21:15:32.654948+0000","last_fullsized":"2023-12-16T21:15:32.654948+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:28:33.936466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":12,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656234+0000","last_change":"2023-12-16T21:15:31.635935+0000","last_active":"2023-12-16T21:15:32.656234+0000","last_peered":"2023-12-16T21:15:32.656234+0000","last_clean":"2023-12-16T21:15:32.656234+0000","last_became_active":"2023-12-16T21:15:31.635557+0000","last_became_peered":"2023-12-16T21:15:31.635557+0000","last_unstale":"2023-12-16T21:15:32.656234+0000","last_undegraded":"2023-12-16T21:15:32.656234+0000","last_fullsized":"2023-12-16T21:15:32.656234+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:07:18.410105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.653022+0000","last_change":"2023-12-16T21:15:32.644054+0000","last_active":"2023-12-16T21:15:32.653022+0000","last_peered":"2023-12-16T21:15:32.653022+0000","last_clean":"2023-12-16T21:15:32.653022+0000","last_became_active":"2023-12-16T21:15:32.643780+0000","last_became_peered":"2023-12-16T21:15:32.643780+0000","last_unstale":"2023-12-16T21:15:32.653022+0000","last_undegraded":"2023-12-16T21:15:32.653022+0000","last_fullsized":"2023-12-16T21:15:32.653022+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:54:35.100969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.640170+0000","last_change":"2023-12-16T21:15:31.637648+0000","last_active":"2023-12-16T21:15:32.640170+0000","last_peered":"2023-12-16T21:15:32.640170+0000","last_clean":"2023-12-16T21:15:32.640170+0000","last_became_active":"2023-12-16T21:15:31.637299+0000","last_became_peered":"2023-12-16T21:15:31.637299+0000","last_unstale":"2023-12-16T21:15:32.640170+0000","last_undegraded":"2023-12-16T21:15:32.640170+0000","last_fullsized":"2023-12-16T21:15:32.640170+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:26:43.428510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.652616+0000","last_change":"2023-12-16T21:15:32.650271+0000","last_active":"2023-12-16T21:15:32.652616+0000","last_peered":"2023-12-16T21:15:32.652616+0000","last_clean":"2023-12-16T21:15:32.652616+0000","last_became_active":"2023-12-16T21:15:32.650029+0000","last_became_peered":"2023-12-16T21:15:32.650029+0000","last_unstale":"2023-12-16T21:15:32.652616+0000","last_undegraded":"2023-12-16T21:15:32.652616+0000","last_fullsized":"2023-12-16T21:15:32.652616+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:45:43.377533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.654240+0000","last_change":"2023-12-16T21:15:31.639259+0000","last_active":"2023-12-16T21:15:32.654240+0000","last_peered":"2023-12-16T21:15:32.654240+0000","last_clean":"2023-12-16T21:15:32.654240+0000","last_became_active":"2023-12-16T21:15:31.638964+0000","last_became_peered":"2023-12-16T21:15:31.638964+0000","last_unstale":"2023-12-16T21:15:32.654240+0000","last_undegraded":"2023-12-16T21:15:32.654240+0000","last_fullsized":"2023-12-16T21:15:32.654240+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:28:38.157109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.649127+0000","last_change":"2023-12-16T21:15:32.648189+0000","last_active":"2023-12-16T21:15:32.649127+0000","last_peered":"2023-12-16T21:15:32.649127+0000","last_clean":"2023-12-16T21:15:32.649127+0000","last_became_active":"2023-12-16T21:15:32.647661+0000","last_became_peered":"2023-12-16T21:15:32.647661+0000","last_unstale":"2023-12-16T21:15:32.649127+0000","last_undegraded":"2023-12-16T21:15:32.649127+0000","last_fullsized":"2023-12-16T21:15:32.649127+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:20:10.477992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.637218+0000","last_change":"2023-12-16T21:15:31.632812+0000","last_active":"2023-12-16T21:15:32.637218+0000","last_peered":"2023-12-16T21:15:32.637218+0000","last_clean":"2023-12-16T21:15:32.637218+0000","last_became_active":"2023-12-16T21:15:31.632110+0000","last_became_peered":"2023-12-16T21:15:31.632110+0000","last_unstale":"2023-12-16T21:15:32.637218+0000","last_undegraded":"2023-12-16T21:15:32.637218+0000","last_fullsized":"2023-12-16T21:15:32.637218+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:31:01.329665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656129+0000","last_change":"2023-12-16T21:15:32.648321+0000","last_active":"2023-12-16T21:15:32.656129+0000","last_peered":"2023-12-16T21:15:32.656129+0000","last_clean":"2023-12-16T21:15:32.656129+0000","last_became_active":"2023-12-16T21:15:32.648022+0000","last_became_peered":"2023-12-16T21:15:32.648022+0000","last_unstale":"2023-12-16T21:15:32.656129+0000","last_undegraded":"2023-12-16T21:15:32.656129+0000","last_fullsized":"2023-12-16T21:15:32.656129+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:35.587180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.630267+0000","last_change":"2023-12-16T21:15:31.642923+0000","last_active":"2023-12-16T21:15:32.630267+0000","last_peered":"2023-12-16T21:15:32.630267+0000","last_clean":"2023-12-16T21:15:32.630267+0000","last_became_active":"2023-12-16T21:15:31.642612+0000","last_became_peered":"2023-12-16T21:15:31.642612+0000","last_unstale":"2023-12-16T21:15:32.630267+0000","last_undegraded":"2023-12-16T21:15:32.630267+0000","last_fullsized":"2023-12-16T21:15:32.630267+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T22:14:00.555170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.655845+0000","last_change":"2023-12-16T21:15:32.648680+0000","last_active":"2023-12-16T21:15:32.655845+0000","last_peered":"2023-12-16T21:15:32.655845+0000","last_clean":"2023-12-16T21:15:32.655845+0000","last_became_active":"2023-12-16T21:15:32.648397+0000","last_became_peered":"2023-12-16T21:15:32.648397+0000","last_unstale":"2023-12-16T21:15:32.655845+0000","last_undegraded":"2023-12-16T21:15:32.655845+0000","last_fullsized":"2023-12-16T21:15:32.655845+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:22:32.537598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.633038+0000","last_change":"2023-12-16T21:15:31.633820+0000","last_active":"2023-12-16T21:15:32.633038+0000","last_peered":"2023-12-16T21:15:32.633038+0000","last_clean":"2023-12-16T21:15:32.633038+0000","last_became_active":"2023-12-16T21:15:31.633526+0000","last_became_peered":"2023-12-16T21:15:31.633526+0000","last_unstale":"2023-12-16T21:15:32.633038+0000","last_undegraded":"2023-12-16T21:15:32.633038+0000","last_fullsized":"2023-12-16T21:15:32.633038+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:59:19.041122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.651255+0000","last_change":"2023-12-16T21:15:32.646932+0000","last_active":"2023-12-16T21:15:32.651255+0000","last_peered":"2023-12-16T21:15:32.651255+0000","last_clean":"2023-12-16T21:15:32.651255+0000","last_became_active":"2023-12-16T21:15:32.646676+0000","last_became_peered":"2023-12-16T21:15:32.646676+0000","last_unstale":"2023-12-16T21:15:32.651255+0000","last_undegraded":"2023-12-16T21:15:32.651255+0000","last_fullsized":"2023-12-16T21:15:32.651255+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":13,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658055+0000","last_change":"2023-12-16T21:15:31.639980+0000","last_active":"2023-12-16T21:15:32.658055+0000","last_peered":"2023-12-16T21:15:32.658055+0000","last_clean":"2023-12-16T21:15:32.658055+0000","last_became_active":"2023-12-16T21:15:31.639651+0000","last_became_peered":"2023-12-16T21:15:31.639651+0000","last_unstale":"2023-12-16T21:15:32.658055+0000","last_undegraded":"2023-12-16T21:15:32.658055+0000","last_fullsized":"2023-12-16T21:15:32.658055+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:53:34.629855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.648167+0000","last_change":"2023-12-16T21:15:32.643658+0000","last_active":"2023-12-16T21:15:32.648167+0000","last_peered":"2023-12-16T21:15:32.648167+0000","last_clean":"2023-12-16T21:15:32.648167+0000","last_became_active":"2023-12-16T21:15:32.643337+0000","last_became_peered":"2023-12-16T21:15:32.643337+0000","last_unstale":"2023-12-16T21:15:32.648167+0000","last_undegraded":"2023-12-16T21:15:32.648167+0000","last_fullsized":"2023-12-16T21:15:32.648167+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:35.587180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.636129+0000","last_change":"2023-12-16T21:15:31.634228+0000","last_active":"2023-12-16T21:15:32.636129+0000","last_peered":"2023-12-16T21:15:32.636129+0000","last_clean":"2023-12-16T21:15:32.636129+0000","last_became_active":"2023-12-16T21:15:31.633943+0000","last_became_peered":"2023-12-16T21:15:31.633943+0000","last_unstale":"2023-12-16T21:15:32.636129+0000","last_undegraded":"2023-12-16T21:15:32.636129+0000","last_fullsized":"2023-12-16T21:15:32.636129+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:20:51.013316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.658551+0000","last_change":"2023-12-16T21:15:32.653418+0000","last_active":"2023-12-16T21:15:32.658551+0000","last_peered":"2023-12-16T21:15:32.658551+0000","last_clean":"2023-12-16T21:15:32.658551+0000","last_became_active":"2023-12-16T21:15:32.653146+0000","last_became_peered":"2023-12-16T21:15:32.653146+0000","last_unstale":"2023-12-16T21:15:32.658551+0000","last_undegraded":"2023-12-16T21:15:32.658551+0000","last_fullsized":"2023-12-16T21:15:32.658551+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T08:31:00.145990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.631812+0000","last_change":"2023-12-16T21:15:31.641639+0000","last_active":"2023-12-16T21:15:32.631812+0000","last_peered":"2023-12-16T21:15:32.631812+0000","last_clean":"2023-12-16T21:15:32.631812+0000","last_became_active":"2023-12-16T21:15:31.641118+0000","last_became_peered":"2023-12-16T21:15:31.641118+0000","last_unstale":"2023-12-16T21:15:32.631812+0000","last_undegraded":"2023-12-16T21:15:32.631812+0000","last_fullsized":"2023-12-16T21:15:32.631812+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:32:57.874803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657037+0000","last_change":"2023-12-16T21:15:32.649554+0000","last_active":"2023-12-16T21:15:32.657037+0000","last_peered":"2023-12-16T21:15:32.657037+0000","last_clean":"2023-12-16T21:15:32.657037+0000","last_became_active":"2023-12-16T21:15:32.649294+0000","last_became_peered":"2023-12-16T21:15:32.649294+0000","last_unstale":"2023-12-16T21:15:32.657037+0000","last_undegraded":"2023-12-16T21:15:32.657037+0000","last_fullsized":"2023-12-16T21:15:32.657037+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T21:42:48.000718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.638090+0000","last_change":"2023-12-16T21:15:31.633388+0000","last_active":"2023-12-16T21:15:32.638090+0000","last_peered":"2023-12-16T21:15:32.638090+0000","last_clean":"2023-12-16T21:15:32.638090+0000","last_became_active":"2023-12-16T21:15:31.633135+0000","last_became_peered":"2023-12-16T21:15:31.633135+0000","last_unstale":"2023-12-16T21:15:32.638090+0000","last_undegraded":"2023-12-16T21:15:32.638090+0000","last_fullsized":"2023-12-16T21:15:32.638090+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T00:34:25.748300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.656448+0000","last_change":"2023-12-16T21:15:32.654941+0000","last_active":"2023-12-16T21:15:32.656448+0000","last_peered":"2023-12-16T21:15:32.656448+0000","last_clean":"2023-12-16T21:15:32.656448+0000","last_became_active":"2023-12-16T21:15:32.654663+0000","last_became_peered":"2023-12-16T21:15:32.654663+0000","last_unstale":"2023-12-16T21:15:32.656448+0000","last_undegraded":"2023-12-16T21:15:32.656448+0000","last_fullsized":"2023-12-16T21:15:32.656448+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:43.155045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.637375+0000","last_change":"2023-12-16T21:15:31.641281+0000","last_active":"2023-12-16T21:15:32.637375+0000","last_peered":"2023-12-16T21:15:32.637375+0000","last_clean":"2023-12-16T21:15:32.637375+0000","last_became_active":"2023-12-16T21:15:31.640470+0000","last_became_peered":"2023-12-16T21:15:31.640470+0000","last_unstale":"2023-12-16T21:15:32.637375+0000","last_undegraded":"2023-12-16T21:15:32.637375+0000","last_fullsized":"2023-12-16T21:15:32.637375+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T03:37:44.577523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.647653+0000","last_change":"2023-12-16T21:15:32.645684+0000","last_active":"2023-12-16T21:15:32.647653+0000","last_peered":"2023-12-16T21:15:32.647653+0000","last_clean":"2023-12-16T21:15:32.647653+0000","last_became_active":"2023-12-16T21:15:32.645394+0000","last_became_peered":"2023-12-16T21:15:32.645394+0000","last_unstale":"2023-12-16T21:15:32.647653+0000","last_undegraded":"2023-12-16T21:15:32.647653+0000","last_fullsized":"2023-12-16T21:15:32.647653+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:12:27.601073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.633906+0000","last_change":"2023-12-16T21:15:31.640752+0000","last_active":"2023-12-16T21:15:32.633906+0000","last_peered":"2023-12-16T21:15:32.633906+0000","last_clean":"2023-12-16T21:15:32.633906+0000","last_became_active":"2023-12-16T21:15:31.640513+0000","last_became_peered":"2023-12-16T21:15:31.640513+0000","last_unstale":"2023-12-16T21:15:32.633906+0000","last_undegraded":"2023-12-16T21:15:32.633906+0000","last_fullsized":"2023-12-16T21:15:32.633906+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:25:42.211225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.657885+0000","last_change":"2023-12-16T21:15:32.648992+0000","last_active":"2023-12-16T21:15:32.657885+0000","last_peered":"2023-12-16T21:15:32.657885+0000","last_clean":"2023-12-16T21:15:32.657885+0000","last_became_active":"2023-12-16T21:15:32.648693+0000","last_became_peered":"2023-12-16T21:15:32.648693+0000","last_unstale":"2023-12-16T21:15:32.657885+0000","last_undegraded":"2023-12-16T21:15:32.657885+0000","last_fullsized":"2023-12-16T21:15:32.657885+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T06:13:30.242051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.634605+0000","last_change":"2023-12-16T21:15:31.634447+0000","last_active":"2023-12-16T21:15:32.634605+0000","last_peered":"2023-12-16T21:15:32.634605+0000","last_clean":"2023-12-16T21:15:32.634605+0000","last_became_active":"2023-12-16T21:15:31.634139+0000","last_became_peered":"2023-12-16T21:15:31.634139+0000","last_unstale":"2023-12-16T21:15:32.634605+0000","last_undegraded":"2023-12-16T21:15:32.634605+0000","last_fullsized":"2023-12-16T21:15:32.634605+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:59:34.577140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.649174+0000","last_change":"2023-12-16T21:15:32.645298+0000","last_active":"2023-12-16T21:15:32.649174+0000","last_peered":"2023-12-16T21:15:32.649174+0000","last_clean":"2023-12-16T21:15:32.649174+0000","last_became_active":"2023-12-16T21:15:32.645031+0000","last_became_peered":"2023-12-16T21:15:32.645031+0000","last_unstale":"2023-12-16T21:15:32.649174+0000","last_undegraded":"2023-12-16T21:15:32.649174+0000","last_fullsized":"2023-12-16T21:15:32.649174+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:31.617258+0000","last_clean_scrub_stamp":"2023-12-16T21:15:31.617258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T04:58:44.030796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-16T21:15:32.631656+0000","last_change":"2023-12-16T21:15:31.633038+0000","last_active":"2023-12-16T21:15:32.631656+0000","last_peered":"2023-12-16T21:15:32.631656+0000","last_clean":"2023-12-16T21:15:32.631656+0000","last_became_active":"2023-12-16T21:15:31.632736+0000","last_became_peered":"2023-12-16T21:15:31.632736+0000","last_unstale":"2023-12-16T21:15:32.631656+0000","last_undegraded":"2023-12-16T21:15:32.631656+0000","last_fullsized":"2023-12-16T21:15:32.631656+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:15:30.607218+0000","last_clean_scrub_stamp":"2023-12-16T21:15:30.607218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:24:42.145005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":190,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-12-16T21:15:31.629355+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:15:31.629355+0000","last_peered":"2023-12-16T21:15:31.629355+0000","last_clean":"2023-12-16T21:15:31.629355+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:15:31.629355+0000","last_undegraded":"2023-12-16T21:15:31.629355+0000","last_fullsized":"2023-12-16T21:15:31.629355+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1175552,"data_stored":830638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67800000000000005,"5min":1.159,"15min":1.159},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69299999999999995,"5min":1.181,"15min":1.181},"last":0.52700000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.624,"15min":0.624},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.749,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.503}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67700000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.627}]},{"osd":6,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.40300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56999999999999995}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48499999999999999,"15min":0.47199999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.755,"15min":0.755},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.505}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.43099999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81399999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.627},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.42099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27580,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.625},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.498},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.70899999999999996,"15min":0.73899999999999999},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46700000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64600000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.39600000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.438},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.51100000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.752,"15min":0.752},"last":0.46700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":585728,"data_stored":240852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.56599999999999995}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58299999999999996}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.621}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.435}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.496},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46500000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54800000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.251},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.33000000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.38400000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.63500000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.66100000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.59199999999999997}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68200000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":593920,"data_stored":241882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68700000000000006,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.433,"15min":0.438},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58499999999999996,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.311},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":1.367,"15min":1.367},"last":0.34399999999999997}]},{"osd":3,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65700000000000003,"5min":1.46,"15min":1.46},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.874,"15min":0.874},"last":0.67300000000000004}]},{"osd":5,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.78000000000000003}]},{"osd":6,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.62,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.76800000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31740,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340100,"statfs":{"total":96636764160,"available":96604262400,"internally_reserved":0,"allocated":569344,"data_stored":233495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47699999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.60399999999999998}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32340,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339500,"statfs":{"total":96636764160,"available":96603648000,"internally_reserved":0,"allocated":1183744,"data_stored":830694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.248}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.37}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46500000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.42299999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.438}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:15:35.997 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-16T21:15:35.997 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-16T21:15:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.147+0000 7f949d6ad640 1 Processor -- start 2023-12-16T21:15:36.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.147+0000 7f949d6ad640 1 -- start start 2023-12-16T21:15:36.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f949d6ad640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94980b43a0 0x7f94980b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f949d6ad640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94980d00f0 con 0x7f94980d3850 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f949d6ad640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94980d03d0 con 0x7f949815d420 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f949d6ad640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94980d1420 con 0x7f94980b43a0 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f94977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94980b43a0 0x7f94980b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f94977fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f94980b43a0 0x7f94980b2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59752/0 (socket says 172.21.15.46:59752) 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.148+0000 7f94977fe640 1 -- 172.21.15.46:0/3639946211 learned_addr learned my addr 172.21.15.46:0/3639946211 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.149+0000 7f94977fe640 1 -- 172.21.15.46:0/3639946211 >> v1:172.21.15.121:6789/0 conn(0x7f949815d420 legacy=0x7f94980d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.149+0000 7f94977fe640 1 -- 172.21.15.46:0/3639946211 >> v1:172.21.15.46:6789/0 conn(0x7f94980d3850 legacy=0x7f94980ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.149+0000 7f94977fe640 1 -- 172.21.15.46:0/3639946211 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980d06b0 con 0x7f94980b43a0 2023-12-16T21:15:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.149+0000 7f94977fe640 1 --2- 172.21.15.46:0/3639946211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94980b43a0 0x7f94980b2b50 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9488009e50 tx=0x7f948802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7c3a48b9bc90ec server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f9495ffb640 1 -- 172.21.15.46:0/3639946211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948802cbb0 con 0x7f94980b43a0 2023-12-16T21:15:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f9495ffb640 1 -- 172.21.15.46:0/3639946211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948802cd10 con 0x7f94980b43a0 2023-12-16T21:15:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f9495ffb640 1 -- 172.21.15.46:0/3639946211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488034610 con 0x7f94980b43a0 2023-12-16T21:15:36.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 -- 172.21.15.46:0/3639946211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94980b43a0 msgr2=0x7f94980b2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 --2- 172.21.15.46:0/3639946211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94980b43a0 0x7f94980b2b50 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9488009e50 tx=0x7f948802c010 comp rx=0 tx=0).stop 2023-12-16T21:15:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 -- 172.21.15.46:0/3639946211 shutdown_connections 2023-12-16T21:15:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 --2- 172.21.15.46:0/3639946211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f94980b43a0 0x7f94980b2b50 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 -- 172.21.15.46:0/3639946211 >> 172.21.15.46:0/3639946211 conn(0x7f949808b000 msgr2=0x7f949809c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 -- 172.21.15.46:0/3639946211 shutdown_connections 2023-12-16T21:15:36.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.150+0000 7f949d6ad640 1 -- 172.21.15.46:0/3639946211 wait complete. 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.151+0000 7f949d6ad640 1 Processor -- start 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 -- start start 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94980b43a0 0x7f94980cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94980d3850 0x7f9498149d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 0x7f94980b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980cd6a0 con 0x7f949815d420 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94980ccb20 con 0x7f94980b43a0 2023-12-16T21:15:36.153 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f949d6ad640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94980cce00 con 0x7f94980d3850 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f94977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 0x7f94980b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f94977fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 0x7f94980b7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59766/0 (socket says 172.21.15.46:59766) 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.152+0000 7f94977fe640 1 -- 172.21.15.46:0/2054649846 learned_addr learned my addr 172.21.15.46:0/2054649846 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f9496ffd640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94980b43a0 0x7f94980cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94967fc640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94980d3850 0x7f9498149d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94977fe640 1 -- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94980d3850 msgr2=0x7f9498149d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94977fe640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94980d3850 0x7f9498149d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94977fe640 1 -- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94980b43a0 msgr2=0x7f94980cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94977fe640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94980b43a0 0x7f94980cc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94977fe640 1 -- 172.21.15.46:0/2054649846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9488009b30 con 0x7f949815d420 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f94977fe640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 0x7f94980b7fa0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9488009e50 tx=0x7f9488004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9488009510 con 0x7f949815d420 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f948803e450 con 0x7f949815d420 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94980cd0e0 con 0x7f949815d420 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f948803e800 con 0x7f949815d420 2023-12-16T21:15:36.154 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.153+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94980cc730 con 0x7f949815d420 2023-12-16T21:15:36.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.154+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94980cc8c0 con 0x7f949815d420 2023-12-16T21:15:36.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.154+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f948803e960 con 0x7f949815d420 2023-12-16T21:15:36.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.155+0000 7f948f7fe640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f947806ae40 0x7f947806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.155+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94880ac450 con 0x7f949815d420 2023-12-16T21:15:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.155+0000 7f9496ffd640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f947806ae40 0x7f947806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.155+0000 7f9496ffd640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f947806ae40 0x7f947806d300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f94980ce0c0 tx=0x7f9490009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:36.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.159+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f9488077640 con 0x7f949815d420 2023-12-16T21:15:36.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.404+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9498069400 con 0x7f949815d420 2023-12-16T21:15:36.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.406+0000 7f948f7fe640 1 -- 172.21.15.46:0/2054649846 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7f948807ac60 con 0x7f949815d420 2023-12-16T21:15:36.406 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:15:36.406 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:15:36.406 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2023-12-16T21:15:36.406 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:15:36.406 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-12-16T21:15:36.406 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f947806ae40 msgr2=0x7f947806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f947806ae40 0x7f947806d300 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f94980ce0c0 tx=0x7f9490009040 comp rx=0 tx=0).stop 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 msgr2=0x7f94980b7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 0x7f94980b7fa0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f9488009e50 tx=0x7f9488004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 shutdown_connections 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f947806ae40 0x7f947806d300 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f949815d420 0x7f94980b7fa0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f94980d3850 0x7f9498149d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 --2- 172.21.15.46:0/2054649846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94980b43a0 0x7f94980cc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 >> 172.21.15.46:0/2054649846 conn(0x7f949808b000 msgr2=0x7f94981617d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 shutdown_connections 2023-12-16T21:15:36.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.408+0000 7f949d6ad640 1 -- 172.21.15.46:0/2054649846 wait complete. 2023-12-16T21:15:36.418 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.525+0000 7f21d47c9640 1 Processor -- start 2023-12-16T21:15:36.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.525+0000 7f21d47c9640 1 -- start start 2023-12-16T21:15:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.526+0000 7f21d47c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21cc0d2910 0x7f21cc0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.526+0000 7f21d47c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21cc0d3730 0x7f21cc0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.526+0000 7f21d47c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0b2b10 con 0x7f21cc15d330 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.526+0000 7f21d47c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc0d3bc0 con 0x7f21cc0d2910 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.526+0000 7f21d47c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc0d1590 con 0x7f21cc0d3730 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.526+0000 7f21d1d64640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21cc0d2910 0x7f21cc0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d1d64640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f21cc0d2910 0x7f21cc0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59776/0 (socket says 172.21.15.46:59776) 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d1d64640 1 -- 172.21.15.46:0/2110762241 learned_addr learned my addr 172.21.15.46:0/2110762241 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d2d66640 1 --2- 172.21.15.46:0/2110762241 >> v2:172.21.15.121:3300/0 conn(0x7f21cc0d3730 0x7f21cc0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d1d64640 1 -- 172.21.15.46:0/2110762241 >> v2:172.21.15.121:3300/0 conn(0x7f21cc0d3730 msgr2=0x7f21cc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d1d64640 1 --2- 172.21.15.46:0/2110762241 >> v2:172.21.15.121:3300/0 conn(0x7f21cc0d3730 0x7f21cc0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d1d64640 1 -- 172.21.15.46:0/2110762241 >> v1:172.21.15.121:6789/0 conn(0x7f21cc15d330 legacy=0x7f21cc0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:36.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.527+0000 7f21d1d64640 1 -- 172.21.15.46:0/2110762241 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0d1870 con 0x7f21cc0d2910 2023-12-16T21:15:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d1d64640 1 --2- 172.21.15.46:0/2110762241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d2910 0x7f21cc0d31f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f21cc06c500 tx=0x7f21bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b95033a59bb9502 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d1563640 1 -- 172.21.15.46:0/2110762241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031870 con 0x7f21cc0d2910 2023-12-16T21:15:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d1563640 1 -- 172.21.15.46:0/2110762241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc0319d0 con 0x7f21cc0d2910 2023-12-16T21:15:36.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d1563640 1 -- 172.21.15.46:0/2110762241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031d60 con 0x7f21cc0d2910 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 -- 172.21.15.46:0/2110762241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d2910 msgr2=0x7f21cc0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2110762241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d2910 0x7f21cc0d31f0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f21cc06c500 tx=0x7f21bc02b910 comp rx=0 tx=0).stop 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 -- 172.21.15.46:0/2110762241 shutdown_connections 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2110762241 >> v2:172.21.15.121:3300/0 conn(0x7f21cc0d3730 0x7f21cc0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2110762241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d2910 0x7f21cc0d31f0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 -- 172.21.15.46:0/2110762241 >> 172.21.15.46:0/2110762241 conn(0x7f21cc072020 msgr2=0x7f21cc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 -- 172.21.15.46:0/2110762241 shutdown_connections 2023-12-16T21:15:36.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.528+0000 7f21d47c9640 1 -- 172.21.15.46:0/2110762241 wait complete. 2023-12-16T21:15:36.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 Processor -- start 2023-12-16T21:15:36.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 -- start start 2023-12-16T21:15:36.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0d2910 0x7f21cc0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 0x7f21cc0c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21cc15d330 0x7f21cc0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0cb840 con 0x7f21cc0d3730 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0cabc0 con 0x7f21cc0d2910 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d47c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0caea0 con 0x7f21cc15d330 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d1d64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 0x7f21cc0c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d1d64640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 0x7f21cc0c8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59780/0 (socket says 172.21.15.46:59780) 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.530+0000 7f21d1d64640 1 -- 172.21.15.46:0/2948897996 learned_addr learned my addr 172.21.15.46:0/2948897996 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d2565640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0d2910 0x7f21cc0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d2d66640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21cc15d330 0x7f21cc0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d1d64640 1 -- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21cc15d330 msgr2=0x7f21cc0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d1d64640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21cc15d330 0x7f21cc0c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d1d64640 1 -- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0d2910 msgr2=0x7f21cc0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d1d64640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0d2910 0x7f21cc0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d1d64640 1 -- 172.21.15.46:0/2948897996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc0094d0 con 0x7f21cc0d3730 2023-12-16T21:15:36.532 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d2565640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0d2910 0x7f21cc0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d1d64640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 0x7f21cc0c8870 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21bc02b950 tx=0x7f21bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc03ac60 con 0x7f21cc0d3730 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc03adc0 con 0x7f21cc0d3730 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc0cb180 con 0x7f21cc0d3730 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0445f0 con 0x7f21cc0d3730 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.531+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc0cbc60 con 0x7f21cc0d3730 2023-12-16T21:15:36.533 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.532+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc0cb980 con 0x7f21cc0d3730 2023-12-16T21:15:36.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.533+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f21bc04d450 con 0x7f21cc0d3730 2023-12-16T21:15:36.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.533+0000 7f21c2ffd640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f21b006ae40 0x7f21b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:36.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.533+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21bc02d040 con 0x7f21cc0d3730 2023-12-16T21:15:36.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.533+0000 7f21d2565640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f21b006ae40 0x7f21b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:36.534 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.534+0000 7f21d2565640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f21b006ae40 0x7f21b006d300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f21c8001660 tx=0x7f21c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:36.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.537+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f21cc0cb980 con 0x7f21cc0d3730 2023-12-16T21:15:36.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.749+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21cc0cb980 con 0x7f21cc0d3730 2023-12-16T21:15:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.749+0000 7f21c2ffd640 1 -- 172.21.15.46:0/2948897996 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f21cc0cb980 con 0x7f21cc0d3730 2023-12-16T21:15:36.749 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:36.749 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:15:31.993483+0000","modified":"2023-12-16T21:15:33.986432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"b","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/2015861199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":2015861199},{"type":"v1","addr":"172.21.15.121:6837","nonce":2015861199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-16T21:15:36.749 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f21b006ae40 msgr2=0x7f21b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f21b006ae40 0x7f21b006d300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f21c8001660 tx=0x7f21c8009040 comp rx=0 tx=0).stop 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 msgr2=0x7f21cc0c8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 0x7f21cc0c8870 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f21bc02b950 tx=0x7f21bc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 shutdown_connections 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f21b006ae40 0x7f21b006d300 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc0d3730 0x7f21cc0c8870 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21cc15d330 0x7f21cc0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 --2- 172.21.15.46:0/2948897996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21cc0d2910 0x7f21cc0ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:36.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 >> 172.21.15.46:0/2948897996 conn(0x7f21cc072020 msgr2=0x7f21cc072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:36.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 shutdown_connections 2023-12-16T21:15:36.752 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:36.751+0000 7f21d47c9640 1 -- 172.21.15.46:0/2948897996 wait complete. 2023-12-16T21:15:37.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-16T21:15:31.993483+0000', 'modified': '2023-12-16T21:15:33.986432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5946}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5946': {'gid': 5946, 'name': 'b', 'rank': 0, 'incarnation': 18, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6837/2015861199', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6835', 'nonce': 2015861199}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 2015861199}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-16T21:15:37.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-16T21:15:37.164 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-16T21:15:37.278 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.279+0000 7f3416ee7640 1 Processor -- start 2023-12-16T21:15:37.279 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.279+0000 7f3416ee7640 1 -- start start 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3416ee7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3410155790 0x7f341006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3416ee7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 0x7f341015ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3416ee7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3410155b70 con 0x7f341006a4d0 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3416ee7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341006ee30 con 0x7f3410155790 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3416ee7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341006f3b0 con 0x7f341015b5e0 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3415484640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f341006a4d0 0x7f34100714a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34484/0 (socket says 172.21.15.46:34484) 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3415484640 1 -- 172.21.15.46:0/139770118 learned_addr learned my addr 172.21.15.46:0/139770118 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:37.282 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f3414c83640 1 --2- 172.21.15.46:0/139770118 >> v2:172.21.15.121:3300/0 conn(0x7f3410155790 0x7f341006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340ffff640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 0x7f341015ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905442290 0 0) 0x7f3410155b70 con 0x7f341006a4d0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3400003520 con 0x7f341006a4d0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4208618669 0 0) 0x7f3400003520 con 0x7f341006a4d0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 msgr2=0x7f341015ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 0x7f341015ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 >> v2:172.21.15.121:3300/0 conn(0x7f3410155790 msgr2=0x7f341006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 --2- 172.21.15.46:0/139770118 >> v2:172.21.15.121:3300/0 conn(0x7f3410155790 0x7f341006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341015c170 con 0x7f341006a4d0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f34080019f0 con 0x7f341006a4d0 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.280+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 >> v1:172.21.15.46:6789/0 conn(0x7f341006a4d0 legacy=0x7f34100714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340f7fe640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 0x7f3400003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340f7fe640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3400003ef0 0x7f34000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340f7fe640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400004d00 0x7f3400005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.283 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341006ee30 con 0x7f3400003ef0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341006f3b0 con 0x7f3400004d00 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f340000b980 con 0x7f341015b5e0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340ffff640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3400003ef0 0x7f34000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f3414c83640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 0x7f3400003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f3415484640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400004d00 0x7f3400005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340ffff640 1 -- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 msgr2=0x7f3400003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340ffff640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015b5e0 0x7f3400003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340ffff640 1 -- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400004d00 msgr2=0x7f3400005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340ffff640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400004d00 0x7f3400005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.281+0000 7f340ffff640 1 -- 172.21.15.46:0/139770118 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3400003520 con 0x7f3400003ef0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f340ffff640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3400003ef0 0x7f34000047c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009ac0 tx=0x7f34000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58950a435f56e379 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc02c6b0 con 0x7f3400003ef0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc02c810 con 0x7f3400003ef0 2023-12-16T21:15:37.284 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f340f7fe640 1 -- 172.21.15.46:0/139770118 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc02cba0 con 0x7f3400003ef0 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f3416ee7640 1 -- 172.21.15.46:0/139770118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3400003ef0 msgr2=0x7f34000047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f3416ee7640 1 --2- 172.21.15.46:0/139770118 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3400003ef0 0x7f34000047c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009ac0 tx=0x7f34000057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f3415484640 1 -- 172.21.15.46:0/139770118 reap_dead start 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f3416ee7640 1 -- 172.21.15.46:0/139770118 shutdown_connections 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.282+0000 7f3416ee7640 1 -- 172.21.15.46:0/139770118 >> 172.21.15.46:0/139770118 conn(0x7f34100b1e50 msgr2=0x7f34100b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.283+0000 7f3416ee7640 1 -- 172.21.15.46:0/139770118 shutdown_connections 2023-12-16T21:15:37.285 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.283+0000 7f3416ee7640 1 -- 172.21.15.46:0/139770118 wait complete. 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.284+0000 7f3416ee7640 1 Processor -- start 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.284+0000 7f3416ee7640 1 -- start start 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400003ef0 0x7f34100b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015aed0 0x7f34101f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 0x7f34100ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34101f2ea0 con 0x7f34101f6e90 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34101f3460 con 0x7f3400003ef0 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34101f3e30 con 0x7f341015aed0 2023-12-16T21:15:37.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 0x7f34100ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 0x7f34100ad960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59792/0 (socket says 172.21.15.46:59792) 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 -- 172.21.15.46:0/2623706670 learned_addr learned my addr 172.21.15.46:0/2623706670 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3414c83640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400003ef0 0x7f34100b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f340ffff640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015aed0 0x7f34101f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 -- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015aed0 msgr2=0x7f34101f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015aed0 0x7f34101f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 -- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400003ef0 msgr2=0x7f34100b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400003ef0 0x7f34100b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 -- 172.21.15.46:0/2623706670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3410159c40 con 0x7f34101f6e90 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3414c83640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400003ef0 0x7f34100b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f340ffff640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015aed0 0x7f34101f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3415484640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 0x7f34100ad960 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f34080013d0 tx=0x7f3408007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f340d7fa640 1 -- 172.21.15.46:0/2623706670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340800ac20 con 0x7f34101f6e90 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f340d7fa640 1 -- 172.21.15.46:0/2623706670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f340800ad80 con 0x7f34101f6e90 2023-12-16T21:15:37.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3410159f20 con 0x7f34101f6e90 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f340d7fa640 1 -- 172.21.15.46:0/2623706670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408013630 con 0x7f34101f6e90 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.285+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34101f7310 con 0x7f34101f6e90 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.287+0000 7f340d7fa640 1 -- 172.21.15.46:0/2623706670 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3408013790 con 0x7f34101f6e90 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.287+0000 7f340d7fa640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f33f406ae40 0x7f33f406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.287+0000 7f340d7fa640 1 -- 172.21.15.46:0/2623706670 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3408089e00 con 0x7f34101f6e90 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.287+0000 7f3414c83640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f33f406ae40 0x7f33f406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.288+0000 7f3414c83640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f33f406ae40 0x7f33f406d300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3404009c40 tx=0x7f3404008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.288+0000 7f3419a7f740 1 Processor -- start 2023-12-16T21:15:37.290 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.288+0000 7f3419a7f740 1 -- start start 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd886b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8b66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8870d0 0x55ecfd8b6c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3419a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd7fa090 con 0x55ecfd8b3d40 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd7f7780 con 0x55ecfd8870d0 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd7f3070 con 0x55ecfd8b3960 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f340ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8b66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f340ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8b66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59794/0 (socket says 172.21.15.46:59794) 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f340ffff640 1 -- 172.21.15.46:0/796351297 learned_addr learned my addr 172.21.15.46:0/796351297 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3415484640 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8870d0 0x55ecfd8b6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.289+0000 7f3414c83640 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd886b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f340ffff640 1 -- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 msgr2=0x55ecfd886b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f340ffff640 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd886b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f340ffff640 1 -- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8870d0 msgr2=0x55ecfd8b6c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f340ffff640 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8870d0 0x55ecfd8b6c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f340ffff640 1 -- 172.21.15.46:0/796351297 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecfd7b5d00 con 0x55ecfd8b3d40 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3414c83640 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd886b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f340ffff640 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8b66f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f33fc02c380 tx=0x7f33fc02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a358deb5b1ce1e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f33fa7fc640 1 -- 172.21.15.46:0/796351297 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc004230 con 0x55ecfd8b3d40 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f33fa7fc640 1 -- 172.21.15.46:0/796351297 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc004390 con 0x55ecfd8b3d40 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f33fa7fc640 1 -- 172.21.15.46:0/796351297 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc02cb80 con 0x55ecfd8b3d40 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 -- 172.21.15.46:0/796351297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 msgr2=0x55ecfd8b66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8b66f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f33fc02c380 tx=0x7f33fc02e000 comp rx=0 tx=0).stop 2023-12-16T21:15:37.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 -- 172.21.15.46:0/796351297 shutdown_connections 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8b66f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8870d0 0x55ecfd8b6c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 --2- 172.21.15.46:0/796351297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd886b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.290+0000 7f3419a7f740 1 -- 172.21.15.46:0/796351297 >> 172.21.15.46:0/796351297 conn(0x55ecfd888e00 msgr2=0x55ecfd8d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.291+0000 7f3419a7f740 1 -- 172.21.15.46:0/796351297 shutdown_connections 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.291+0000 7f3419a7f740 1 -- 172.21.15.46:0/796351297 wait complete. 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.291+0000 7f3419a7f740 1 -- start start 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.291+0000 7f3419a7f740 1 Processor -- start 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8e1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 0x55ecfd8b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd7ace70 con 0x55ecfd8aef60 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd7acb10 con 0x55ecfd8b3d40 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd7722e0 con 0x55ecfd8b3960 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3419a7f740 10 client.0 _ll_register_callbacks cb 0x55ecfd504ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:15:37.298 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 0x55ecfd8b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 0x55ecfd8b5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59810/0 (socket says 172.21.15.46:59810) 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 -- 172.21.15.46:0/1228556041 learned_addr learned my addr 172.21.15.46:0/1228556041 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3414c83640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f340ffff640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8e1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 -- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 msgr2=0x55ecfd8ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8ae370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 -- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 msgr2=0x55ecfd8e1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8e1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.292+0000 7f3415484640 1 -- 172.21.15.46:0/1228556041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3400003520 con 0x55ecfd8aef60 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3415484640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 0x55ecfd8b5490 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3408088290 tx=0x7f34080593a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3e93623b8582bd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f33ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f33ef7fe640 1 -- 172.21.15.46:0/1228556041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408070030 con 0x55ecfd8aef60 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f33ef7fe640 1 -- 172.21.15.46:0/1228556041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3408060bd0 con 0x55ecfd8aef60 2023-12-16T21:15:37.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ecfd7ebf00 con 0x55ecfd8aef60 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f33ef7fe640 1 -- 172.21.15.46:0/1228556041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3408088080 con 0x55ecfd8aef60 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 10 client.6072 fetch_fsmap learned FSMap version 20 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55ecfd7f16f0 con 0x55ecfd8aef60 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f33ef7fe640 1 -- 172.21.15.46:0/1228556041 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408074070 con 0x55ecfd8aef60 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f33ef7fe640 1 -- 172.21.15.46:0/1228556041 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 20) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3408012070 con 0x55ecfd8aef60 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 10 client.6072 fetch_fsmap finished waiting for FSMap version 20 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 10 client.6072 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 20 client.6072 populate_metadata read hostname 'smithi046' 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8ec820 0x55ecfd8eec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 4 client.6072 mds_command: new command op to 5946 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 --> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecfd7ebc20 con 0x55ecfd8ec820 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.293+0000 7f3414c83640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8ec820 0x55ecfd8eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.294+0000 7f33ef7fe640 10 client.6072 ms_handle_connect on v2:172.21.15.121:6835/2015861199 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.294+0000 7f3414c83640 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8ec820 0x55ecfd8eec10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c192dea81c8b0f2 server_cookie=3ca63efc1a4a2c00 in_seq=0 out_seq=0 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f33ef7fe640 1 -- 172.21.15.46:0/1228556041 <== mds.0 v2:172.21.15.121:6835/2015861199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ecfd7ebc20 con 0x55ecfd8ec820 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f33ef7fe640 10 client.6072 handle_command_reply: tid=0 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 client.6072 shutdown 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 msgr2=0x55ecfd8b5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 0x55ecfd8b5490 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3408088290 tx=0x7f34080593a0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 shutdown_connections 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8ec820 0x55ecfd8eec10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8aef60 0x55ecfd8b5490 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8e1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 --2- 172.21.15.46:0/1228556041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f33ef7fe640 0 client.6072 ms_handle_reset on v2:172.21.15.121:6835/2015861199 2023-12-16T21:15:37.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 >> 172.21.15.46:0/1228556041 conn(0x55ecfd8870d0 msgr2=0x55ecfd8d4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 shutdown_connections 2023-12-16T21:15:37.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 1 -- 172.21.15.46:0/1228556041 wait complete. 2023-12-16T21:15:37.302 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.295+0000 7f3419a7f740 20 client.6072 trim_cache size 0 max 16384 2023-12-16T21:15:37.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 Processor -- start 2023-12-16T21:15:37.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 -- start start 2023-12-16T21:15:37.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8aef60 0x55ecfd8d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8862b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8867f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd8ba3c0 con 0x55ecfd8b3d40 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd8b5ee0 con 0x55ecfd8aef60 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.311+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd8b41b0 con 0x55ecfd8b3960 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8867f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8867f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59822/0 (socket says 172.21.15.46:59822) 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 -- 172.21.15.46:0/4139582876 learned_addr learned my addr 172.21.15.46:0/4139582876 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f340ffff640 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8862b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3414c83640 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8aef60 0x55ecfd8d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 -- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 msgr2=0x55ecfd8862b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8862b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 -- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8aef60 msgr2=0x55ecfd8d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8aef60 0x55ecfd8d1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 -- 172.21.15.46:0/4139582876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecfd78ed70 con 0x55ecfd8b3d40 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f3415484640 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8867f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3408058aa0 tx=0x7f340806e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8420876656b2a88e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f33faffd640 1 -- 172.21.15.46:0/4139582876 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408070050 con 0x55ecfd8b3d40 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f33faffd640 1 -- 172.21.15.46:0/4139582876 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408074070 con 0x55ecfd8b3d40 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.312+0000 7f33faffd640 1 -- 172.21.15.46:0/4139582876 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408088080 con 0x55ecfd8b3d40 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 -- 172.21.15.46:0/4139582876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 msgr2=0x55ecfd8867f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8867f0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f3408058aa0 tx=0x7f340806e000 comp rx=0 tx=0).stop 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 -- 172.21.15.46:0/4139582876 shutdown_connections 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3d40 0x55ecfd8867f0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8b3960 0x55ecfd8862b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 --2- 172.21.15.46:0/4139582876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8aef60 0x55ecfd8d1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 -- 172.21.15.46:0/4139582876 >> 172.21.15.46:0/4139582876 conn(0x55ecfd8870d0 msgr2=0x55ecfd8ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 -- 172.21.15.46:0/4139582876 shutdown_connections 2023-12-16T21:15:37.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 -- 172.21.15.46:0/4139582876 wait complete. 2023-12-16T21:15:37.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 -- start start 2023-12-16T21:15:37.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.313+0000 7f3419a7f740 1 Processor -- start 2023-12-16T21:15:37.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8aef60 0x55ecfd93ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 0x55ecfd88c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd88c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.321 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd8aed90 con 0x55ecfd8b3960 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd93b6c0 con 0x55ecfd8b3d40 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecfd8aeab0 con 0x55ecfd8aef60 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3419a7f740 10 client.0 _ll_register_callbacks cb 0x7f33fc037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f340ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 0x55ecfd88c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f340ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 0x55ecfd88c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59836/0 (socket says 172.21.15.46:59836) 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f340ffff640 1 -- 172.21.15.46:0/3090105381 learned_addr learned my addr 172.21.15.46:0/3090105381 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3414c83640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8aef60 0x55ecfd93ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.314+0000 7f3415484640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd88c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f340ffff640 1 -- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8aef60 msgr2=0x55ecfd93ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f340ffff640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8aef60 0x55ecfd93ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f340ffff640 1 -- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 msgr2=0x55ecfd88c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f340ffff640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd88c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f340ffff640 1 -- 172.21.15.46:0/3090105381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecfd7f16f0 con 0x55ecfd8b3960 2023-12-16T21:15:37.322 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f340ffff640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 0x55ecfd88c3c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009a90 tx=0x7f33fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1699e6904978f01b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f33ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f33ef7fe640 1 -- 172.21.15.46:0/3090105381 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc02cd90 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f33ef7fe640 1 -- 172.21.15.46:0/3090105381 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33fc003610 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33fc004390 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f33ef7fe640 1 -- 172.21.15.46:0/3090105381 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33fc004500 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 10 client.6078 fetch_fsmap learned FSMap version 20 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x55ecfd7b4560 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f33ef7fe640 1 -- 172.21.15.46:0/3090105381 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33fc004390 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f33ef7fe640 1 -- 172.21.15.46:0/3090105381 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 20) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f33fc003810 con 0x55ecfd8b3960 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 10 client.6078 fetch_fsmap finished waiting for FSMap version 20 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 10 client.6078 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 20 client.6078 populate_metadata read hostname 'smithi046' 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8c73b0 0x55ecfd8c7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 4 client.6078 mds_command: new command op to 5946 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.315+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 --> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ecfd4f36c0 con 0x55ecfd8c73b0 2023-12-16T21:15:37.323 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3414c83640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8c73b0 0x55ecfd8c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3414c83640 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8c73b0 0x55ecfd8c7790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e02c3514ebf48d81 server_cookie=2047edb3236df327 in_seq=0 out_seq=0 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f33ef7fe640 10 client.6078 ms_handle_connect on v2:172.21.15.121:6835/2015861199 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f33ef7fe640 1 -- 172.21.15.46:0/3090105381 <== mds.0 v2:172.21.15.121:6835/2015861199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ecfd4f36c0 con 0x55ecfd8c73b0 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f33ef7fe640 10 client.6078 handle_command_reply: tid=0 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 client.6078 shutdown 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 msgr2=0x55ecfd88c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 0x55ecfd88c3c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009a90 tx=0x7f33fc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 shutdown_connections 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:6835/2015861199,v1:172.21.15.121:6837/2015861199] conn(0x55ecfd8c73b0 0x55ecfd8c7790 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ecfd8b3960 0x55ecfd88c3c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ecfd8b3d40 0x55ecfd88c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 --2- 172.21.15.46:0/3090105381 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ecfd8aef60 0x55ecfd93ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 >> 172.21.15.46:0/3090105381 conn(0x55ecfd8870d0 msgr2=0x55ecfd8d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f33ef7fe640 0 client.6078 ms_handle_reset on v2:172.21.15.121:6835/2015861199 2023-12-16T21:15:37.324 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.316+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 shutdown_connections 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3419a7f740 1 -- 172.21.15.46:0/3090105381 wait complete. 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3419a7f740 20 client.6078 trim_cache size 0 max 16384 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "2397e6ee-7ffe-4438-b275-b4e10dea2d2f", 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 5946, 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2023-12-16T21:15:37.325 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 4.6895204560000003, 2023-12-16T21:15:37.326 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 20, 2023-12-16T21:15:37.326 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 50, 2023-12-16T21:15:37.326 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 50, 2023-12-16T21:15:37.326 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 10.686357875000001 2023-12-16T21:15:37.326 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:15:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f33f406ae40 msgr2=0x7f33f406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f33f406ae40 0x7f33f406d300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3404009c40 tx=0x7f3404008040 comp rx=0 tx=0).stop 2023-12-16T21:15:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 msgr2=0x7f34100ad960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.327 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 0x7f34100ad960 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f34080013d0 tx=0x7f3408007bd0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 shutdown_connections 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f33f406ae40 0x7f33f406d300 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f34101f6e90 0x7f34100ad960 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f341015aed0 0x7f34101f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 --2- 172.21.15.46:0/2623706670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3400003ef0 0x7f34100b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 >> 172.21.15.46:0/2623706670 conn(0x7f34100b1e50 msgr2=0x7f341015df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 shutdown_connections 2023-12-16T21:15:37.328 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.317+0000 7f3416ee7640 1 -- 172.21.15.46:0/2623706670 wait complete. 2023-12-16T21:15:37.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:15:37.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.435+0000 7ff8adf89640 1 Processor -- start 2023-12-16T21:15:37.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.435+0000 7ff8adf89640 1 -- start start 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8adf89640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8a806ea50 0x7ff8a806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8adf89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a8161d20 con 0x7ff8a80d0ed0 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8adf89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a806f370 con 0x7ff8a8160fb0 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8adf89640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a815e760 con 0x7ff8a806ea50 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8a806ea50 0x7ff8a806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8a77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8a806ea50 0x7ff8a806ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59840/0 (socket says 172.21.15.46:59840) 2023-12-16T21:15:37.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8a77fe640 1 -- 172.21.15.46:0/2970761402 learned_addr learned my addr 172.21.15.46:0/2970761402 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8a77fe640 1 -- 172.21.15.46:0/2970761402 >> v1:172.21.15.121:6789/0 conn(0x7ff8a8160fb0 legacy=0x7ff8a8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:37.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8a77fe640 1 -- 172.21.15.46:0/2970761402 >> v1:172.21.15.46:6789/0 conn(0x7ff8a80d0ed0 legacy=0x7ff8a80d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.437 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.436+0000 7ff8a77fe640 1 -- 172.21.15.46:0/2970761402 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a815d9a0 con 0x7ff8a806ea50 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.437+0000 7ff8a77fe640 1 --2- 172.21.15.46:0/2970761402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a806ea50 0x7ff8a806ee30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff898009870 tx=0x7ff89802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8171e46132cbf75c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.437+0000 7ff8a67fc640 1 -- 172.21.15.46:0/2970761402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898031870 con 0x7ff8a806ea50 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.437+0000 7ff8a67fc640 1 -- 172.21.15.46:0/2970761402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8980319d0 con 0x7ff8a806ea50 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.437+0000 7ff8a67fc640 1 -- 172.21.15.46:0/2970761402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff898031d60 con 0x7ff8a806ea50 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 -- 172.21.15.46:0/2970761402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a806ea50 msgr2=0x7ff8a806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 --2- 172.21.15.46:0/2970761402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a806ea50 0x7ff8a806ee30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff898009870 tx=0x7ff89802b910 comp rx=0 tx=0).stop 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 -- 172.21.15.46:0/2970761402 shutdown_connections 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 --2- 172.21.15.46:0/2970761402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a806ea50 0x7ff8a806ee30 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.438 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 -- 172.21.15.46:0/2970761402 >> 172.21.15.46:0/2970761402 conn(0x7ff8a806a6b0 msgr2=0x7ff8a806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 -- 172.21.15.46:0/2970761402 shutdown_connections 2023-12-16T21:15:37.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.438+0000 7ff8adf89640 1 -- 172.21.15.46:0/2970761402 wait complete. 2023-12-16T21:15:37.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.439+0000 7ff8adf89640 1 Processor -- start 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.439+0000 7ff8adf89640 1 -- start start 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8adf89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a806ea50 0x7ff8a8159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8adf89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a80d0ed0 0x7ff8a815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8adf89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 0x7ff8a81568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8adf89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80ca370 con 0x7ff8a8160fb0 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8adf89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a813f640 con 0x7ff8a806ea50 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8adf89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a815a4b0 con 0x7ff8a80d0ed0 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 0x7ff8a81568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 0x7ff8a81568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59846/0 (socket says 172.21.15.46:59846) 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 -- 172.21.15.46:0/72029561 learned_addr learned my addr 172.21.15.46:0/72029561 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a77fe640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a806ea50 0x7ff8a8159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a6ffd640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a80d0ed0 0x7ff8a815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 -- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a80d0ed0 msgr2=0x7ff8a815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a80d0ed0 0x7ff8a815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 -- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a806ea50 msgr2=0x7ff8a8159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a806ea50 0x7ff8a8159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.440+0000 7ff8a7fff640 1 -- 172.21.15.46:0/72029561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8980094d0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.441+0000 7ff8a7fff640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 0x7ff8a81568f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000c730 tx=0x7ff8a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.441+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0003b70 con 0x7ff8a8160fb0 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.441+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a0003cf0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.441+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a80ca5c0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.441+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0003ea0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.441+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a80cab80 con 0x7ff8a8160fb0 2023-12-16T21:15:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.442+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a80cad10 con 0x7ff8a8160fb0 2023-12-16T21:15:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.442+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff8a0005310 con 0x7ff8a8160fb0 2023-12-16T21:15:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.443+0000 7ff88ffff640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff89406ae40 0x7ff89406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.443+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8a00891e0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.443+0000 7ff8a77fe640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff89406ae40 0x7ff89406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:37.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.443+0000 7ff8a77fe640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff89406ae40 0x7ff89406d300 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff898036000 tx=0x7ff89803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:37.446 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.447+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff8a00543d0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.610 INFO:tasks.ceph.mds.b.smithi121.stderr:2023-12-16T21:15:37.610+0000 7f2dd5b09640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:15:37.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.664+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff8a8070ac0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.664 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.664+0000 7ff88ffff640 1 -- 172.21.15.46:0/72029561 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff8a00579f0 con 0x7ff8a8160fb0 2023-12-16T21:15:37.664 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:15:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff89406ae40 msgr2=0x7ff89406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff89406ae40 0x7ff89406d300 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff898036000 tx=0x7ff89803c040 comp rx=0 tx=0).stop 2023-12-16T21:15:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 msgr2=0x7ff8a81568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:37.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 0x7ff8a81568f0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000c730 tx=0x7ff8a000cbf0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 shutdown_connections 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff89406ae40 0x7ff89406d300 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8a8160fb0 0x7ff8a81568f0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8a80d0ed0 0x7ff8a815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 --2- 172.21.15.46:0/72029561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a806ea50 0x7ff8a8159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.666+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 >> 172.21.15.46:0/72029561 conn(0x7ff8a806a6b0 msgr2=0x7ff8a80d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.667+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 shutdown_connections 2023-12-16T21:15:37.667 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:37.667+0000 7ff8adf89640 1 -- 172.21.15.46:0/72029561 wait complete. 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:15:37.677 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:37.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:37.678 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:15:37.693 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:15:37.693 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:15:37.694 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42547sec preferred_lft 42547sec 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:37.695 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link 2023-12-16T21:15:37.696 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.696 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:37.696 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:37.696 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::20ee:6cff:fee1:5615/64 scope link 2023-12-16T21:15:37.696 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:37.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:37.696 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:15:37.763 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:15:37.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:37.764 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:15:37.820 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:15:37.820 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:15:37.821 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:15:37.821 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:15:37.821 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:15:37.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:37.821 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:15:37.821 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:15:37.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:15:37.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:15:37.821 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:15:37.821 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:15:37.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:37.926 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:15:37.941 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:15:37.941 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:15:37.941 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:37.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:37.942 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:15:38.012 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:15:38.012 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:38.012 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:38.068 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:38.068 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:38.068 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:38.123 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:38.194 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:38.251 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:38.252 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:38.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:38.254 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:38.306 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:15:38.306 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:15:38.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:38.380 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:38.436 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:38.436 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:38.436 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:38.437 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:38.438 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:38.439 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:38.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:38.439 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:38.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:15:38.443+0000 7f2b4e20d480 -1 init, newargv = 0x55596445e770 newargc=15 2023-12-16T21:15:38.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[109562]: starting ceph client 2023-12-16T21:15:38.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[109562]: starting fuse 2023-12-16T21:15:38.459 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:15:38.460 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:15:38.460 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:38.461 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:15:38.462 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:15:38.585 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.109562.asok 2023-12-16T21:15:38.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:15:38.595 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.109562.asok 2023-12-16T21:15:38.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:38.595 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.109562.asok status 2023-12-16T21:15:38.738 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "109562", 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:15:38.739 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:15:38.740 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6087, 2023-12-16T21:15:38.740 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:15:38.740 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:15:38.740 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:15:38.740 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6087 2023-12-16T21:15:38.740 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:38.741 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:38.741 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:38.741 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:38.741 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2617267241 2023-12-16T21:15:38.741 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:38.741 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2617267241 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6087 192.168.0.1:0/2617267241", 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/2617267241", 2023-12-16T21:15:38.742 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:15:38.743 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 20, 2023-12-16T21:15:38.743 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 50, 2023-12-16T21:15:38.743 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:15:38.743 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:15:38.743 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:15:38.743 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:15:38.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:38.747 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:15:38.747 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:38.762 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:15:38.763 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:38.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:38.763 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:38.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:38.835 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2023-12-16T21:15:38.897 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2023-12-16T21:15:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.050+0000 7fb93b451640 1 Processor -- start 2023-12-16T21:15:39.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.050+0000 7fb93b451640 1 -- start start 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb93b451640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb9340cef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb93b451640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb9340b3400 0x7fb93406c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb93b451640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb934161170 con 0x7fb934146d70 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb93b451640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb934161450 con 0x7fb9340b3400 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb93b451640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb934161730 con 0x7fb934146010 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb9399ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb9340b3400 0x7fb93406c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.051+0000 7fb9391ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb9340cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9391ed640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb9340cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:58984/0 (socket says 172.21.15.46:58984) 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9399ee640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb9340b3400 0x7fb93406c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:33786/0 (socket says 172.21.15.46:33786) 2023-12-16T21:15:39.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9391ed640 1 -- 172.21.15.46:0/1719831907 learned_addr learned my addr 172.21.15.46:0/1719831907 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9389ec640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb934146d70 0x7fb9340b2cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57218/0 (socket says 172.21.15.46:57218) 2023-12-16T21:15:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb92bfff640 1 -- 172.21.15.46:0/1719831907 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558147684 0 0) 0x7fb934161170 con 0x7fb934146d70 2023-12-16T21:15:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9391ed640 1 -- 172.21.15.46:0/1719831907 >> v2:172.21.15.121:3300/0 conn(0x7fb9340b3400 msgr2=0x7fb93406c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:39.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9391ed640 1 --2- 172.21.15.46:0/1719831907 >> v2:172.21.15.121:3300/0 conn(0x7fb9340b3400 0x7fb93406c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9391ed640 1 -- 172.21.15.46:0/1719831907 >> v1:172.21.15.121:6789/0 conn(0x7fb934146d70 legacy=0x7fb9340b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.052+0000 7fb9391ed640 1 -- 172.21.15.46:0/1719831907 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb934161a10 con 0x7fb934146010 2023-12-16T21:15:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb9391ed640 1 --2- 172.21.15.46:0/1719831907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb9340cef50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009870 tx=0x7fb92c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c99606fd8af51d0b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb92bfff640 1 -- 172.21.15.46:0/1719831907 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c030840 con 0x7fb934146010 2023-12-16T21:15:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb92bfff640 1 -- 172.21.15.46:0/1719831907 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92c0309a0 con 0x7fb934146010 2023-12-16T21:15:39.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb92bfff640 1 -- 172.21.15.46:0/1719831907 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92c030d30 con 0x7fb934146010 2023-12-16T21:15:39.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 -- 172.21.15.46:0/1719831907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 msgr2=0x7fb9340cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.058 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 --2- 172.21.15.46:0/1719831907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb9340cef50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb92c009870 tx=0x7fb92c02b980 comp rx=0 tx=0).stop 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 -- 172.21.15.46:0/1719831907 shutdown_connections 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 --2- 172.21.15.46:0/1719831907 >> v2:172.21.15.121:3300/0 conn(0x7fb9340b3400 0x7fb93406c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 --2- 172.21.15.46:0/1719831907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb9340cef50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 -- 172.21.15.46:0/1719831907 >> 172.21.15.46:0/1719831907 conn(0x7fb93406a4d0 msgr2=0x7fb93406a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.053+0000 7fb93b451640 1 -- 172.21.15.46:0/1719831907 shutdown_connections 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.054+0000 7fb93b451640 1 -- 172.21.15.46:0/1719831907 wait complete. 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 Processor -- start 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 -- start start 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9340b3400 0x7fb93413d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb934152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 0x7fb9341f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93415fb10 con 0x7fb934146d70 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb934152fd0 con 0x7fb9340b3400 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.055+0000 7fb93b451640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb93415ef20 con 0x7fb934146010 2023-12-16T21:15:39.059 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 0x7fb9341f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 0x7fb9341f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59848/0 (socket says 172.21.15.46:59848) 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 -- 172.21.15.46:0/1404411794 learned_addr learned my addr 172.21.15.46:0/1404411794 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9389ec640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb934152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9391ed640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9340b3400 0x7fb93413d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 -- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 msgr2=0x7fb934152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb934152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 -- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9340b3400 msgr2=0x7fb93413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9340b3400 0x7fb93413d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 -- 172.21.15.46:0/1404411794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0094d0 con 0x7fb934146d70 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb9399ee640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 0x7fb9341f3270 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb93000efc0 tx=0x7fb93000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb930009ca0 con 0x7fb934146d70 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93000f040 con 0x7fb934146d70 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb934146570 con 0x7fb934146d70 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9300146c0 con 0x7fb934146d70 2023-12-16T21:15:39.060 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.056+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93415f450 con 0x7fb934146d70 2023-12-16T21:15:39.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.058+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb934146be0 con 0x7fb934146d70 2023-12-16T21:15:39.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.058+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb930014820 con 0x7fb934146d70 2023-12-16T21:15:39.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.058+0000 7fb929ffb640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb91806ae40 0x7fb91806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.058+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb93008a350 con 0x7fb934146d70 2023-12-16T21:15:39.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.059+0000 7fb9391ed640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb91806ae40 0x7fb91806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.061 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.059+0000 7fb9391ed640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb91806ae40 0x7fb91806d300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb92c036000 tx=0x7fb92c008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.062 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.063+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb930055540 con 0x7fb934146d70 2023-12-16T21:15:39.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.306+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) v1 -- 0x7fb93415fc50 con 0x7fb934146d70 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.308+0000 7fb929ffb640 1 -- 172.21.15.46:0/1404411794 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v20) v1 ==== 183+0+140 (secure 0 0 0) 0x7fb93415fc50 con 0x7fb934146d70 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow r pool=cephfs_data" 2023-12-16T21:15:39.308 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb91806ae40 msgr2=0x7fb91806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb91806ae40 0x7fb91806d300 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb92c036000 tx=0x7fb92c008030 comp rx=0 tx=0).stop 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 msgr2=0x7fb9341f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 0x7fb9341f3270 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb93000efc0 tx=0x7fb93000c490 comp rx=0 tx=0).stop 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 shutdown_connections 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb91806ae40 0x7fb91806d300 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb934146d70 0x7fb9341f3270 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb934146010 0x7fb934152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 --2- 172.21.15.46:0/1404411794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9340b3400 0x7fb93413d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 >> 172.21.15.46:0/1404411794 conn(0x7fb93406a4d0 msgr2=0x7fb93406cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 shutdown_connections 2023-12-16T21:15:39.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.310+0000 7fb93b451640 1 -- 172.21.15.46:0/1404411794 wait complete. 2023-12-16T21:15:39.320 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:39.321 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:39.336 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:39.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.483+0000 7ff44c89b640 1 Processor -- start 2023-12-16T21:15:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.483+0000 7ff44c89b640 1 -- start start 2023-12-16T21:15:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff44c89b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4480d2910 0x7ff4480d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff44c89b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480b2b10 con 0x7ff44815d330 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff44c89b640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4480d1590 con 0x7ff4480d3730 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff44c89b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4480d3bc0 con 0x7ff4480d2910 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff44659c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff44815d330 0x7ff4480b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34508/0 (socket says 172.21.15.46:34508) 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff44659c640 1 -- 172.21.15.46:0/1859955922 learned_addr learned my addr 172.21.15.46:0/1859955922 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.484+0000 7ff445d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff4480d2910 0x7ff4480d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.485+0000 7ff445d9b640 1 -- 172.21.15.46:0/1859955922 >> v1:172.21.15.121:6789/0 conn(0x7ff4480d3730 legacy=0x7ff4480b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.485+0000 7ff445d9b640 1 -- 172.21.15.46:0/1859955922 >> v1:172.21.15.46:6789/0 conn(0x7ff44815d330 legacy=0x7ff4480b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.485+0000 7ff445d9b640 1 -- 172.21.15.46:0/1859955922 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4480d1870 con 0x7ff4480d2910 2023-12-16T21:15:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.485+0000 7ff445d9b640 1 --2- 172.21.15.46:0/1859955922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d2910 0x7ff4480d31f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff4300097a0 tx=0x7ff43002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264b6bed2a082416 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44559a640 1 -- 172.21.15.46:0/1859955922 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43002e070 con 0x7ff4480d2910 2023-12-16T21:15:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44559a640 1 -- 172.21.15.46:0/1859955922 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4300315a0 con 0x7ff4480d2910 2023-12-16T21:15:39.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44559a640 1 -- 172.21.15.46:0/1859955922 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430031930 con 0x7ff4480d2910 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 -- 172.21.15.46:0/1859955922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d2910 msgr2=0x7ff4480d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 --2- 172.21.15.46:0/1859955922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d2910 0x7ff4480d31f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff4300097a0 tx=0x7ff43002b840 comp rx=0 tx=0).stop 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 -- 172.21.15.46:0/1859955922 shutdown_connections 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 --2- 172.21.15.46:0/1859955922 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d2910 0x7ff4480d31f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 -- 172.21.15.46:0/1859955922 >> 172.21.15.46:0/1859955922 conn(0x7ff448072020 msgr2=0x7ff448072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 -- 172.21.15.46:0/1859955922 shutdown_connections 2023-12-16T21:15:39.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.486+0000 7ff44c89b640 1 -- 172.21.15.46:0/1859955922 wait complete. 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.487+0000 7ff44c89b640 1 Processor -- start 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 -- start start 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4480d2910 0x7ff4480cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 0x7ff4480ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff44815d330 0x7ff44815aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481e66e0 con 0x7ff4480d3730 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4480cc4e0 con 0x7ff4480d2910 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff44c89b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4480cb960 con 0x7ff44815d330 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff445d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 0x7ff4480ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff445d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 0x7ff4480ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59864/0 (socket says 172.21.15.46:59864) 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.488+0000 7ff445d9b640 1 -- 172.21.15.46:0/429839146 learned_addr learned my addr 172.21.15.46:0/429839146 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:39.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff44659c640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4480d2910 0x7ff4480cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff445d9b640 1 -- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff44815d330 msgr2=0x7ff44815aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff445d9b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff44815d330 0x7ff44815aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff445d9b640 1 -- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4480d2910 msgr2=0x7ff4480cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff445d9b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4480d2910 0x7ff4480cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff445d9b640 1 -- 172.21.15.46:0/429839146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff430009400 con 0x7ff4480d3730 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff445d9b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 0x7ff4480ccbb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff430034040 tx=0x7ff4300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430032900 con 0x7ff4480d3730 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff430032a80 con 0x7ff4480d3730 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4480cbc40 con 0x7ff4480d3730 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430032e30 con 0x7ff4480d3730 2023-12-16T21:15:39.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.489+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4480cc200 con 0x7ff4480d3730 2023-12-16T21:15:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.490+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4480c6c10 con 0x7ff4480d3730 2023-12-16T21:15:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.490+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff43003b3f0 con 0x7ff4480d3730 2023-12-16T21:15:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.491+0000 7ff43effd640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff42806ae40 0x7ff42806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.491+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff4300ab710 con 0x7ff4480d3730 2023-12-16T21:15:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.491+0000 7ff44659c640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff42806ae40 0x7ff42806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.492+0000 7ff44659c640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff42806ae40 0x7ff42806d300 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff438005c00 tx=0x7ff438009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.495+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff4480c6c10 con 0x7ff4480d3730 2023-12-16T21:15:39.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.675+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4480c6c10 con 0x7ff4480d3730 2023-12-16T21:15:39.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.676+0000 7ff43effd640 1 -- 172.21.15.46:0/429839146 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7ff4480c6c10 con 0x7ff4480d3730 2023-12-16T21:15:39.676 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:39.676 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:39.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff42806ae40 msgr2=0x7ff42806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff42806ae40 0x7ff42806d300 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff438005c00 tx=0x7ff438009040 comp rx=0 tx=0).stop 2023-12-16T21:15:39.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 msgr2=0x7ff4480ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 0x7ff4480ccbb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff430034040 tx=0x7ff4300046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 shutdown_connections 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff42806ae40 0x7ff42806d300 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4480d3730 0x7ff4480ccbb0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff44815d330 0x7ff44815aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 --2- 172.21.15.46:0/429839146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4480d2910 0x7ff4480cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 >> 172.21.15.46:0/429839146 conn(0x7ff448072020 msgr2=0x7ff4480b6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 shutdown_connections 2023-12-16T21:15:39.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.678+0000 7ff44c89b640 1 -- 172.21.15.46:0/429839146 wait complete. 2023-12-16T21:15:39.689 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:15:29.641456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:15:30.976721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:39.689 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:39.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 Processor -- start 2023-12-16T21:15:39.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 -- start start 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 0x7f6ec81618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6ec81409d0 0x7f6ec806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec81406f0 con 0x7f6ec8140310 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec8161e20 con 0x7f6ec81409d0 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.799+0000 7f6ece361640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8143960 con 0x7f6ec806abe0 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 0x7f6ec81618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ecc8fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6ec81409d0 0x7f6ec806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec7fff640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6ec8140310 0x7f6ec806a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57242/0 (socket says 172.21.15.46:57242) 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec77fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 0x7f6ec81618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:59018/0 (socket says 172.21.15.46:59018) 2023-12-16T21:15:39.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec7fff640 1 -- 172.21.15.46:0/647986817 learned_addr learned my addr 172.21.15.46:0/647986817 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:39.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec77fe640 1 -- 172.21.15.46:0/647986817 >> v2:172.21.15.121:3300/0 conn(0x7f6ec81409d0 msgr2=0x7f6ec806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:39.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec77fe640 1 --2- 172.21.15.46:0/647986817 >> v2:172.21.15.121:3300/0 conn(0x7f6ec81409d0 0x7f6ec806efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec77fe640 1 -- 172.21.15.46:0/647986817 >> v1:172.21.15.121:6789/0 conn(0x7f6ec8140310 legacy=0x7f6ec806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec77fe640 1 -- 172.21.15.46:0/647986817 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8142de0 con 0x7f6ec806abe0 2023-12-16T21:15:39.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ec6ffd640 1 -- 172.21.15.46:0/647986817 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082558661 0 0) 0x7f6ec81406f0 con 0x7f6ec8140310 2023-12-16T21:15:39.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.800+0000 7f6ecc8fe640 1 --2- 172.21.15.46:0/647986817 >> v2:172.21.15.121:3300/0 conn(0x7f6ec81409d0 0x7f6ec806efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:39.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.801+0000 7f6ec77fe640 1 --2- 172.21.15.46:0/647986817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 0x7f6ec81618e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc0097a0 tx=0x7f6ebc02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8843154a374b5cb4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.801+0000 7f6ec6ffd640 1 -- 172.21.15.46:0/647986817 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc031780 con 0x7f6ec806abe0 2023-12-16T21:15:39.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.801+0000 7f6ec6ffd640 1 -- 172.21.15.46:0/647986817 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc0318e0 con 0x7f6ec806abe0 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.801+0000 7f6ec6ffd640 1 -- 172.21.15.46:0/647986817 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc031c70 con 0x7f6ec806abe0 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 -- 172.21.15.46:0/647986817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 msgr2=0x7f6ec81618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 --2- 172.21.15.46:0/647986817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 0x7f6ec81618e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc0097a0 tx=0x7f6ebc02b820 comp rx=0 tx=0).stop 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 -- 172.21.15.46:0/647986817 shutdown_connections 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 --2- 172.21.15.46:0/647986817 >> v2:172.21.15.121:3300/0 conn(0x7f6ec81409d0 0x7f6ec806efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 --2- 172.21.15.46:0/647986817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec806abe0 0x7f6ec81618e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 -- 172.21.15.46:0/647986817 >> 172.21.15.46:0/647986817 conn(0x7f6ec80b1e50 msgr2=0x7f6ec80b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 -- 172.21.15.46:0/647986817 shutdown_connections 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.802+0000 7f6ece361640 1 -- 172.21.15.46:0/647986817 wait complete. 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.803+0000 7f6ece361640 1 Processor -- start 2023-12-16T21:15:39.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.803+0000 7f6ece361640 1 -- start start 2023-12-16T21:15:39.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ece361640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 0x7f6ec81f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ece361640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ec8140310 0x7f6ec81f6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ece361640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec81409d0 0x7f6ec8145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ece361640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec81467f0 con 0x7f6ec806abe0 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ece361640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec812f5a0 con 0x7f6ec8140310 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ece361640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8146230 con 0x7f6ec81409d0 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 0x7f6ec81f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 0x7f6ec81f60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59868/0 (socket says 172.21.15.46:59868) 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 -- 172.21.15.46:0/1522047650 learned_addr learned my addr 172.21.15.46:0/1522047650 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ecc8fe640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec81409d0 0x7f6ec8145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec77fe640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ec8140310 0x7f6ec81f6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 -- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec81409d0 msgr2=0x7f6ec8145590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec81409d0 0x7f6ec8145590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 -- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ec8140310 msgr2=0x7f6ec81f6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ec8140310 0x7f6ec81f6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.804+0000 7f6ec7fff640 1 -- 172.21.15.46:0/1522047650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec0009a70 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.805+0000 7f6ec7fff640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 0x7f6ec81f60c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800c4d0 tx=0x7f6eb800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.805+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb80074e0 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.805+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb8007660 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.805+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8004a20 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.805+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc009400 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.805+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec812f230 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.806+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec812f090 con 0x7f6ec806abe0 2023-12-16T21:15:39.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.806+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6eb8004b80 con 0x7f6ec806abe0 2023-12-16T21:15:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.806+0000 7f6ec4ff9640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6eb406ae40 0x7f6eb406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.807+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6eb8088810 con 0x7f6ec806abe0 2023-12-16T21:15:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.807+0000 7f6ec77fe640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6eb406ae40 0x7f6eb406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:39.808 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.807+0000 7f6ec77fe640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6eb406ae40 0x7f6eb406d300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc036000 tx=0x7f6ebc008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:39.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.811+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6ec812f090 con 0x7f6ec806abe0 2023-12-16T21:15:39.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.986+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ec812f090 con 0x7f6ec806abe0 2023-12-16T21:15:39.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.986+0000 7f6ec4ff9640 1 -- 172.21.15.46:0/1522047650 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ec812f090 con 0x7f6ec806abe0 2023-12-16T21:15:39.986 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:39.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6eb406ae40 msgr2=0x7f6eb406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6eb406ae40 0x7f6eb406d300 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc036000 tx=0x7f6ebc008030 comp rx=0 tx=0).stop 2023-12-16T21:15:39.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 msgr2=0x7f6ec81f60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:39.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 0x7f6ec81f60c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800c4d0 tx=0x7f6eb800c990 comp rx=0 tx=0).stop 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 shutdown_connections 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6eb406ae40 0x7f6eb406d300 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ec81409d0 0x7f6ec8145590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ec8140310 0x7f6ec81f6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 --2- 172.21.15.46:0/1522047650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ec806abe0 0x7f6ec81f60c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 >> 172.21.15.46:0/1522047650 conn(0x7f6ec80b1e50 msgr2=0x7f6ec80714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.988+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 shutdown_connections 2023-12-16T21:15:39.989 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:39.989+0000 7f6ece361640 1 -- 172.21.15.46:0/1522047650 wait complete. 2023-12-16T21:15:39.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.108+0000 7f4aa8533640 1 Processor -- start 2023-12-16T21:15:40.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.108+0000 7f4aa8533640 1 -- start start 2023-12-16T21:15:40.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa8533640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4aa00d2910 0x7f4aa00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa8533640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4aa00d3730 0x7f4aa00b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa8533640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa00b2b10 con 0x7f4aa015d330 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa8533640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa00d3bc0 con 0x7f4aa00d2910 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa8533640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa00d1590 con 0x7f4aa00d3730 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa5ace640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4aa00d2910 0x7f4aa00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa5ace640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4aa00d2910 0x7f4aa00d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59874/0 (socket says 172.21.15.46:59874) 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.109+0000 7f4aa5ace640 1 -- 172.21.15.46:0/71671181 learned_addr learned my addr 172.21.15.46:0/71671181 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa6ad0640 1 --2- 172.21.15.46:0/71671181 >> v2:172.21.15.121:3300/0 conn(0x7f4aa00d3730 0x7f4aa00b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa5ace640 1 -- 172.21.15.46:0/71671181 >> v2:172.21.15.121:3300/0 conn(0x7f4aa00d3730 msgr2=0x7f4aa00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa5ace640 1 --2- 172.21.15.46:0/71671181 >> v2:172.21.15.121:3300/0 conn(0x7f4aa00d3730 0x7f4aa00b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa5ace640 1 -- 172.21.15.46:0/71671181 >> v1:172.21.15.121:6789/0 conn(0x7f4aa015d330 legacy=0x7f4aa00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa5ace640 1 -- 172.21.15.46:0/71671181 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa00d1870 con 0x7f4aa00d2910 2023-12-16T21:15:40.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa6ad0640 1 --2- 172.21.15.46:0/71671181 >> v2:172.21.15.121:3300/0 conn(0x7f4aa00d3730 0x7f4aa00b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:40.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.110+0000 7f4aa5ace640 1 --2- 172.21.15.46:0/71671181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa00d2910 0x7f4aa00d31f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4aa006c500 tx=0x7f4a9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f843ef0186f4abba server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa52cd640 1 -- 172.21.15.46:0/71671181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94031870 con 0x7f4aa00d2910 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa52cd640 1 -- 172.21.15.46:0/71671181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a940319d0 con 0x7f4aa00d2910 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa52cd640 1 -- 172.21.15.46:0/71671181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94031d60 con 0x7f4aa00d2910 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 -- 172.21.15.46:0/71671181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa00d2910 msgr2=0x7f4aa00d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 --2- 172.21.15.46:0/71671181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa00d2910 0x7f4aa00d31f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4aa006c500 tx=0x7f4a9402b910 comp rx=0 tx=0).stop 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 -- 172.21.15.46:0/71671181 shutdown_connections 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 --2- 172.21.15.46:0/71671181 >> v2:172.21.15.121:3300/0 conn(0x7f4aa00d3730 0x7f4aa00b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 --2- 172.21.15.46:0/71671181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa00d2910 0x7f4aa00d31f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 -- 172.21.15.46:0/71671181 >> 172.21.15.46:0/71671181 conn(0x7f4aa0072020 msgr2=0x7f4aa0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 -- 172.21.15.46:0/71671181 shutdown_connections 2023-12-16T21:15:40.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.111+0000 7f4aa8533640 1 -- 172.21.15.46:0/71671181 wait complete. 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.112+0000 7f4aa8533640 1 Processor -- start 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 -- start start 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aa00d2910 0x7f4aa00cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4aa00d3730 0x7f4aa00c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 0x7f4aa0153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00c6cd0 con 0x7f4aa015d330 2023-12-16T21:15:40.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00cc380 con 0x7f4aa00d2910 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa8533640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa00cc660 con 0x7f4aa00d3730 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa6ad0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 0x7f4aa0153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa6ad0640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 0x7f4aa0153290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59880/0 (socket says 172.21.15.46:59880) 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.113+0000 7f4aa6ad0640 1 -- 172.21.15.46:0/2810051977 learned_addr learned my addr 172.21.15.46:0/2810051977 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa5ace640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4aa00d3730 0x7f4aa00c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa62cf640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aa00d2910 0x7f4aa00cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa6ad0640 1 -- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4aa00d3730 msgr2=0x7f4aa00c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa6ad0640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4aa00d3730 0x7f4aa00c8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa6ad0640 1 -- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aa00d2910 msgr2=0x7f4aa00cbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa6ad0640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aa00d2910 0x7f4aa00cbd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa6ad0640 1 -- 172.21.15.46:0/2810051977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a940094d0 con 0x7f4aa015d330 2023-12-16T21:15:40.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa6ad0640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 0x7f4aa0153290 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4a90007ef0 tx=0x7f4a9000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a90003b90 con 0x7f4aa015d330 2023-12-16T21:15:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9000f040 con 0x7f4aa015d330 2023-12-16T21:15:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000de80 con 0x7f4aa015d330 2023-12-16T21:15:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa00cc940 con 0x7f4aa015d330 2023-12-16T21:15:40.116 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.114+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa00c86c0 con 0x7f4aa015d330 2023-12-16T21:15:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.115+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa00c6e40 con 0x7f4aa015d330 2023-12-16T21:15:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.116+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4a90004030 con 0x7f4aa015d330 2023-12-16T21:15:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.116+0000 7f4a9effd640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4a8c06ae70 0x7f4a8c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.116+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a900898b0 con 0x7f4aa015d330 2023-12-16T21:15:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.116+0000 7f4aa62cf640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4a8c06ae70 0x7f4a8c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.117+0000 7f4aa62cf640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4a8c06ae70 0x7f4a8c06d330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4a98000f60 tx=0x7f4a98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.120+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4aa00c6e40 con 0x7f4aa015d330 2023-12-16T21:15:40.296 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.297+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4aa00c6e10 con 0x7f4aa015d330 2023-12-16T21:15:40.297 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.297+0000 7f4a9effd640 1 -- 172.21.15.46:0/2810051977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a9001a0a0 con 0x7f4aa015d330 2023-12-16T21:15:40.297 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4a8c06ae70 msgr2=0x7f4a8c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4a8c06ae70 0x7f4a8c06d330 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4a98000f60 tx=0x7f4a98009040 comp rx=0 tx=0).stop 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 msgr2=0x7f4aa0153290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 0x7f4aa0153290 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4a90007ef0 tx=0x7f4a9000d4a0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 shutdown_connections 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4a8c06ae70 0x7f4a8c06d330 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4aa015d330 0x7f4aa0153290 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4aa00d3730 0x7f4aa00c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 --2- 172.21.15.46:0/2810051977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4aa00d2910 0x7f4aa00cbd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 >> 172.21.15.46:0/2810051977 conn(0x7f4aa0072020 msgr2=0x7f4aa00b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 shutdown_connections 2023-12-16T21:15:40.299 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.299+0000 7f4aa8533640 1 -- 172.21.15.46:0/2810051977 wait complete. 2023-12-16T21:15:40.309 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:40.418 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.419+0000 7f01f7389640 1 Processor -- start 2023-12-16T21:15:40.419 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.419+0000 7f01f7389640 1 -- start start 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f7389640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 0x7f01f006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f7389640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01f00aeb80 0x7f01f006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f7389640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f00adae0 con 0x7f01f00ad640 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f7389640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00addc0 con 0x7f01f00aeb80 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f7389640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00ae0a0 con 0x7f01f015d1d0 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5125640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 0x7f01f006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01f00aeb80 0x7f01f006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01f00aeb80 0x7f01f006ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:33870/0 (socket says 172.21.15.46:33870) 2023-12-16T21:15:40.420 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f4924640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f01f00ad640 0x7f01f0160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:57250/0 (socket says 172.21.15.46:57250) 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5125640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 0x7f01f006b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:59032/0 (socket says 172.21.15.46:59032) 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 -- 172.21.15.46:0/2211422658 learned_addr learned my addr 172.21.15.46:0/2211422658 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 -- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 msgr2=0x7f01f006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 --2- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 0x7f01f006b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 -- 172.21.15.46:0/2211422658 >> v1:172.21.15.121:6789/0 conn(0x7f01f00ad640 legacy=0x7f01f0160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01e7fff640 1 -- 172.21.15.46:0/2211422658 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210124819 0 0) 0x7f01f00adae0 con 0x7f01f00ad640 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5926640 1 -- 172.21.15.46:0/2211422658 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f00ae380 con 0x7f01f00aeb80 2023-12-16T21:15:40.421 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.420+0000 7f01f5125640 1 --2- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 0x7f01f006b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.421+0000 7f01f5926640 1 --2- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00aeb80 0x7f01f006ba20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009e50 tx=0x7f01e802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10a8afbc22b05315 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.421+0000 7f01e7fff640 1 -- 172.21.15.46:0/2211422658 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e802cb00 con 0x7f01f00aeb80 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.421+0000 7f01e7fff640 1 -- 172.21.15.46:0/2211422658 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e802cc60 con 0x7f01f00aeb80 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.421+0000 7f01e7fff640 1 -- 172.21.15.46:0/2211422658 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e80343f0 con 0x7f01f00aeb80 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 -- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00aeb80 msgr2=0x7f01f006ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 --2- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00aeb80 0x7f01f006ba20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009e50 tx=0x7f01e802bf60 comp rx=0 tx=0).stop 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 -- 172.21.15.46:0/2211422658 shutdown_connections 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 --2- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00aeb80 0x7f01f006ba20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 --2- 172.21.15.46:0/2211422658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f015d1d0 0x7f01f006b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 -- 172.21.15.46:0/2211422658 >> 172.21.15.46:0/2211422658 conn(0x7f01f0070e70 msgr2=0x7f01f0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 -- 172.21.15.46:0/2211422658 shutdown_connections 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.422+0000 7f01f7389640 1 -- 172.21.15.46:0/2211422658 wait complete. 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.423+0000 7f01f7389640 1 Processor -- start 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.423+0000 7f01f7389640 1 -- start start 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f7389640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00ad640 0x7f01f01442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f7389640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00aeb80 0x7f01f01476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f7389640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 0x7f01f01520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f7389640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f015e080 con 0x7f01f015d1d0 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f7389640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f0151c90 con 0x7f01f00ad640 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f7389640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f015dac0 con 0x7f01f00aeb80 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 0x7f01f01520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 0x7f01f01520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59886/0 (socket says 172.21.15.46:59886) 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 -- 172.21.15.46:0/1406676079 learned_addr learned my addr 172.21.15.46:0/1406676079 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:40.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f4924640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00aeb80 0x7f01f01476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5125640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00ad640 0x7f01f01442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 -- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00aeb80 msgr2=0x7f01f01476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00aeb80 0x7f01f01476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 -- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00ad640 msgr2=0x7f01f01442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00ad640 0x7f01f01442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5926640 1 -- 172.21.15.46:0/1406676079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ec009400 con 0x7f01f015d1d0 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.424+0000 7f01f5125640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00ad640 0x7f01f01442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.425+0000 7f01f5926640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 0x7f01f01520b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009e50 tx=0x7f01e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.425+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e8009510 con 0x7f01f015d1d0 2023-12-16T21:15:40.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.425+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e803e450 con 0x7f01f015d1d0 2023-12-16T21:15:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.425+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e8009b30 con 0x7f01f015d1d0 2023-12-16T21:15:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.425+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e803e800 con 0x7f01f015d1d0 2023-12-16T21:15:40.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.425+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f01518b0 con 0x7f01f015d1d0 2023-12-16T21:15:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.426+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f0151dd0 con 0x7f01f015d1d0 2023-12-16T21:15:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.426+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f01e803e960 con 0x7f01f015d1d0 2023-12-16T21:15:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.427+0000 7f01e5ffb640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d406ae40 0x7f01d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.427+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01e80ac3d0 con 0x7f01f015d1d0 2023-12-16T21:15:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.427+0000 7f01f5125640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d406ae40 0x7f01d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.429 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.427+0000 7f01f5125640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d406ae40 0x7f01d406d300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009bb0 tx=0x7f01ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.431+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f01f0151dd0 con 0x7f01f015d1d0 2023-12-16T21:15:40.608 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.609+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f01f0151dd0 con 0x7f01f015d1d0 2023-12-16T21:15:40.609 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.609+0000 7f01e5ffb640 1 -- 172.21.15.46:0/1406676079 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f01f0151dd0 con 0x7f01f015d1d0 2023-12-16T21:15:40.609 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d406ae40 msgr2=0x7f01d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d406ae40 0x7f01d406d300 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009bb0 tx=0x7f01ec008040 comp rx=0 tx=0).stop 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 msgr2=0x7f01f01520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 0x7f01f01520b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009e50 tx=0x7f01e8004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 shutdown_connections 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d406ae40 0x7f01d406d300 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d1d0 0x7f01f01520b0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00aeb80 0x7f01f01476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 --2- 172.21.15.46:0/1406676079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f00ad640 0x7f01f01442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 >> 172.21.15.46:0/1406676079 conn(0x7f01f0070e70 msgr2=0x7f01f006d770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 shutdown_connections 2023-12-16T21:15:40.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.611+0000 7f01f7389640 1 -- 172.21.15.46:0/1406676079 wait complete. 2023-12-16T21:15:40.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:40.731 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.731+0000 7f08d1456640 1 Processor -- start 2023-12-16T21:15:40.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.731+0000 7f08d1456640 1 -- start start 2023-12-16T21:15:40.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08d1456640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08cc0d1d50 0x7f08cc06bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08d1456640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc06ce00 con 0x7f08cc15d6c0 2023-12-16T21:15:40.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08d1456640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc06c1f0 con 0x7f08cc15d2e0 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08d1456640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08cc06c4d0 con 0x7f08cc0d1d50 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08cb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08cc0d1d50 0x7f08cc06bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08ca7fc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f08cc15d6c0 0x7f08cc06b4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56408/0 (socket says 172.21.15.46:56408) 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08ca7fc640 1 -- 172.21.15.46:0/3908353106 learned_addr learned my addr 172.21.15.46:0/3908353106 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.732+0000 7f08cb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08cc0d1d50 0x7f08cc06bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37146/0 (socket says 172.21.15.46:37146) 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.733+0000 7f08cb7fe640 1 -- 172.21.15.46:0/3908353106 >> v1:172.21.15.121:6789/0 conn(0x7f08cc15d2e0 legacy=0x7f08cc0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.733+0000 7f08cb7fe640 1 -- 172.21.15.46:0/3908353106 >> v1:172.21.15.46:6789/0 conn(0x7f08cc15d6c0 legacy=0x7f08cc06b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.733+0000 7f08c9ffb640 1 -- 172.21.15.46:0/3908353106 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509507283 0 0) 0x7f08cc06ce00 con 0x7f08cc15d6c0 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.733+0000 7f08cb7fe640 1 -- 172.21.15.46:0/3908353106 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc06c7b0 con 0x7f08cc0d1d50 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.733+0000 7f08cb7fe640 1 --2- 172.21.15.46:0/3908353106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc0d1d50 0x7f08cc06bbd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009d90 tx=0x7f08c402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc7b2df8043ce02 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08c9ffb640 1 -- 172.21.15.46:0/3908353106 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c402cc60 con 0x7f08cc0d1d50 2023-12-16T21:15:40.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08c9ffb640 1 -- 172.21.15.46:0/3908353106 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c402cdc0 con 0x7f08cc0d1d50 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08c9ffb640 1 -- 172.21.15.46:0/3908353106 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4034660 con 0x7f08cc0d1d50 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 -- 172.21.15.46:0/3908353106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc0d1d50 msgr2=0x7f08cc06bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 --2- 172.21.15.46:0/3908353106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc0d1d50 0x7f08cc06bbd0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009d90 tx=0x7f08c402beb0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 -- 172.21.15.46:0/3908353106 shutdown_connections 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 --2- 172.21.15.46:0/3908353106 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc0d1d50 0x7f08cc06bbd0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 -- 172.21.15.46:0/3908353106 >> 172.21.15.46:0/3908353106 conn(0x7f08cc09c580 msgr2=0x7f08cc0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 -- 172.21.15.46:0/3908353106 shutdown_connections 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.734+0000 7f08d1456640 1 -- 172.21.15.46:0/3908353106 wait complete. 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 Processor -- start 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 -- start start 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08cc0d1d50 0x7f08cc151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08cc15d2e0 0x7f08cc0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 0x7f08cc155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc151dc0 con 0x7f08cc15d6c0 2023-12-16T21:15:40.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc151f70 con 0x7f08cc15d2e0 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.736+0000 7f08d1456640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc152810 con 0x7f08cc0d1d50 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 0x7f08cc155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 0x7f08cc155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37162/0 (socket says 172.21.15.46:37162) 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 -- 172.21.15.46:0/4055882713 learned_addr learned my addr 172.21.15.46:0/4055882713 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08caffd640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08cc0d1d50 0x7f08cc151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08ca7fc640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08cc15d2e0 0x7f08cc0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 -- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08cc0d1d50 msgr2=0x7f08cc151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08cc0d1d50 0x7f08cc151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 -- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08cc15d2e0 msgr2=0x7f08cc0b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08cc15d2e0 0x7f08cc0b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 -- 172.21.15.46:0/4055882713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c4009a70 con 0x7f08cc15d6c0 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08ca7fc640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08cc15d2e0 0x7f08cc0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08cb7fe640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 0x7f08cc155f60 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009eb0 tx=0x7f08c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4003b90 con 0x7f08cc15d6c0 2023-12-16T21:15:40.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c4003cf0 con 0x7f08cc15d6c0 2023-12-16T21:15:40.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc0b8a40 con 0x7f08cc15d6c0 2023-12-16T21:15:40.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4004230 con 0x7f08cc15d6c0 2023-12-16T21:15:40.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.737+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc1524a0 con 0x7f08cc15d6c0 2023-12-16T21:15:40.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.738+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc1529a0 con 0x7f08cc15d6c0 2023-12-16T21:15:40.742 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.739+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f08c403b050 con 0x7f08cc15d6c0 2023-12-16T21:15:40.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.739+0000 7f08b37fe640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f08b806ae40 0x7f08b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:40.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.739+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08c40abf20 con 0x7f08cc15d6c0 2023-12-16T21:15:40.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.740+0000 7f08caffd640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f08b806ae40 0x7f08b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:40.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.740+0000 7f08caffd640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f08b806ae40 0x7f08b806d300 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08bc000f60 tx=0x7f08bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:40.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.744+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f08c4076fd0 con 0x7f08cc15d6c0 2023-12-16T21:15:40.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.933+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f08cc067420 con 0x7f08cc15d6c0 2023-12-16T21:15:40.933 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.933+0000 7f08b37fe640 1 -- 172.21.15.46:0/4055882713 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7f08c4039080 con 0x7f08cc15d6c0 2023-12-16T21:15:40.933 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:40.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f08b806ae40 msgr2=0x7f08b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f08b806ae40 0x7f08b806d300 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08bc000f60 tx=0x7f08bc009040 comp rx=0 tx=0).stop 2023-12-16T21:15:40.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 msgr2=0x7f08cc155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:40.935 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 0x7f08cc155f60 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009eb0 tx=0x7f08c4004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 shutdown_connections 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f08b806ae40 0x7f08b806d300 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08cc15d6c0 0x7f08cc155f60 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08cc15d2e0 0x7f08cc0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 --2- 172.21.15.46:0/4055882713 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08cc0d1d50 0x7f08cc151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 >> 172.21.15.46:0/4055882713 conn(0x7f08cc09c580 msgr2=0x7f08cc0d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 shutdown_connections 2023-12-16T21:15:40.936 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:40.935+0000 7f08d1456640 1 -- 172.21.15.46:0/4055882713 wait complete. 2023-12-16T21:15:40.946 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:40.946 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:40.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.108+0000 7fc002f45640 1 Processor -- start 2023-12-16T21:15:41.108 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.108+0000 7fc002f45640 1 -- start start 2023-12-16T21:15:41.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc002f45640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbffc06ea50 0x7fbffc06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.111 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc002f45640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 0x7fbffc0d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc002f45640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbffc161d20 con 0x7fbffc160fb0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc002f45640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbffc06f370 con 0x7fbffc06ea50 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc002f45640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc15e760 con 0x7fbffc0d0ed0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fbffc160fb0 0x7fbffc161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56416/0 (socket says 172.21.15.46:56416) 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffbfff640 1 -- 172.21.15.46:0/1974895452 learned_addr learned my addr 172.21.15.46:0/1974895452 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3431677218 0 0) 0x7fbffc161d20 con 0x7fbffc160fb0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfec003520 con 0x7fbffc160fb0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc0014e2640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 0x7fbffc0d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc000ce1640 1 --2- 172.21.15.46:0/1974895452 >> v2:172.21.15.121:3300/0 conn(0x7fbffc06ea50 0x7fbffc06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 308743475 0 0) 0x7fbfec003520 con 0x7fbffc160fb0 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 msgr2=0x7fbffc0d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 0x7fbffc0d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.112 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 >> v2:172.21.15.121:3300/0 conn(0x7fbffc06ea50 msgr2=0x7fbffc06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 --2- 172.21.15.46:0/1974895452 >> v2:172.21.15.121:3300/0 conn(0x7fbffc06ea50 0x7fbffc06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc15d9a0 con 0x7fbffc160fb0 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fbfe8001670 con 0x7fbffc160fb0 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.109+0000 7fc000ce1640 1 --2- 172.21.15.46:0/1974895452 >> v2:172.21.15.121:3300/0 conn(0x7fbffc06ea50 0x7fbffc06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 >> v1:172.21.15.46:6789/0 conn(0x7fbffc160fb0 legacy=0x7fbffc161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbfec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 0x7fbfec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec004d00 0x7fbfec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc06f370 con 0x7fbfec003970 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc15e760 con 0x7fbfec004d00 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec00b980 con 0x7fbffc0d0ed0 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc000ce1640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbfec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fbffbfff640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 0x7fbfec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc0014e2640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec004d00 0x7fbfec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc000ce1640 1 -- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 msgr2=0x7fbfec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc000ce1640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc0d0ed0 0x7fbfec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc000ce1640 1 -- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec004d00 msgr2=0x7fbfec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc000ce1640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec004d00 0x7fbfec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc000ce1640 1 -- 172.21.15.46:0/1974895452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec003520 con 0x7fbfec003970 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.110+0000 7fc0014e2640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfec004d00 0x7fbfec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc000ce1640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbfec003d90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbff00098d0 tx=0x7fbfec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ffd3f86d6dd8d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0004230 con 0x7fbfec003970 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0004390 con 0x7fbfec003970 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fbffb7fe640 1 -- 172.21.15.46:0/1974895452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0003830 con 0x7fbfec003970 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc002f45640 1 -- 172.21.15.46:0/1974895452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 msgr2=0x7fbfec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc002f45640 1 --2- 172.21.15.46:0/1974895452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbfec003d90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fbff00098d0 tx=0x7fbfec0057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc0014e2640 1 -- 172.21.15.46:0/1974895452 reap_dead start 2023-12-16T21:15:41.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc002f45640 1 -- 172.21.15.46:0/1974895452 shutdown_connections 2023-12-16T21:15:41.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc002f45640 1 -- 172.21.15.46:0/1974895452 >> 172.21.15.46:0/1974895452 conn(0x7fbffc06a6b0 msgr2=0x7fbffc06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:41.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc002f45640 1 -- 172.21.15.46:0/1974895452 shutdown_connections 2023-12-16T21:15:41.115 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.111+0000 7fc002f45640 1 -- 172.21.15.46:0/1974895452 wait complete. 2023-12-16T21:15:41.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.113+0000 7fc002f45640 1 Processor -- start 2023-12-16T21:15:41.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.113+0000 7fc002f45640 1 -- start start 2023-12-16T21:15:41.117 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbffc13fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbffc0cc450 0x7fbffc158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc15b970 0x7fbffc15bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc1f2cd0 con 0x7fbfec003970 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc15c330 con 0x7fbffc0cc450 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbffc159450 con 0x7fbffc15b970 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbffc13fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbffc13fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37178/0 (socket says 172.21.15.46:37178) 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 -- 172.21.15.46:0/3718349196 learned_addr learned my addr 172.21.15.46:0/3718349196 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc0014e2640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc15b970 0x7fbffc15bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fbffbfff640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbffc0cc450 0x7fbffc158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 -- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc15b970 msgr2=0x7fbffc15bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc15b970 0x7fbffc15bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 -- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbffc0cc450 msgr2=0x7fbffc158f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbffc0cc450 0x7fbffc158f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.118 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 -- 172.21.15.46:0/3718349196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbffc1f6700 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fbffbfff640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbffc0cc450 0x7fbffc158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc000ce1640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbffc13fb70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fbff00099f0 tx=0x7fbff0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0042600 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff0042760 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbffc1f69e0 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0042b10 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.114+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbffc1f6fa0 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.115+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbffc069400 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.116+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbff00373f0 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.116+0000 7fbff97fa640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbfe006ae40 0x7fbfe006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.116+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbff00ab530 con 0x7fbfec003970 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.116+0000 7fbffbfff640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbfe006ae40 0x7fbfe006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.119 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.117+0000 7fbffbfff640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbfe006ae40 0x7fbfe006d300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007fd0 tx=0x7fbfe8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.120 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.120+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fbff00765f0 con 0x7fbfec003970 2023-12-16T21:15:41.300 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.300+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbffc070d30 con 0x7fbfec003970 2023-12-16T21:15:41.301 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.301+0000 7fbff97fa640 1 -- 172.21.15.46:0/3718349196 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fbff0006c30 con 0x7fbfec003970 2023-12-16T21:15:41.301 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:41.301 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbfe006ae40 msgr2=0x7fbfe006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbfe006ae40 0x7fbfe006d300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fbfe8007fd0 tx=0x7fbfe8009040 comp rx=0 tx=0).stop 2023-12-16T21:15:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 msgr2=0x7fbffc13fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbffc13fb70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fbff00099f0 tx=0x7fbff0004730 comp rx=0 tx=0).stop 2023-12-16T21:15:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 shutdown_connections 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbfe006ae40 0x7fbfe006d300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbffc15b970 0x7fbffc15bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbffc0cc450 0x7fbffc158f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 --2- 172.21.15.46:0/3718349196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbfec003970 0x7fbffc13fb70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 >> 172.21.15.46:0/3718349196 conn(0x7fbffc06a6b0 msgr2=0x7fbffc15c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 shutdown_connections 2023-12-16T21:15:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.303+0000 7fc002f45640 1 -- 172.21.15.46:0/3718349196 wait complete. 2023-12-16T21:15:41.314 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:15:29.641456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:15:30.976721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:41.314 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.424+0000 7fe6f36bb640 1 Processor -- start 2023-12-16T21:15:41.423 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.424+0000 7fe6f36bb640 1 -- start start 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f36bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec06abe0 0x7fe6ec1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f36bb640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec1406f0 con 0x7fe6ec140310 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f36bb640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6ec143960 con 0x7fe6ec1409d0 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f36bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec161e20 con 0x7fe6ec06abe0 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f1457640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe6ec140310 0x7fe6ec06a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56424/0 (socket says 172.21.15.46:56424) 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f1457640 1 -- 172.21.15.46:0/1823010011 learned_addr learned my addr 172.21.15.46:0/1823010011 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f0c56640 1 --2- 172.21.15.46:0/1823010011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec06abe0 0x7fe6ec1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6e3fff640 1 -- 172.21.15.46:0/1823010011 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684067978 0 0) 0x7fe6ec1406f0 con 0x7fe6ec140310 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6e3fff640 1 -- 172.21.15.46:0/1823010011 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6d8003520 con 0x7fe6ec140310 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f0c56640 1 -- 172.21.15.46:0/1823010011 >> v1:172.21.15.121:6789/0 conn(0x7fe6ec1409d0 legacy=0x7fe6ec06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.425 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6e3fff640 1 -- 172.21.15.46:0/1823010011 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494080215 0 0) 0x7fe6ec143960 con 0x7fe6ec1409d0 2023-12-16T21:15:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f0c56640 1 -- 172.21.15.46:0/1823010011 >> v1:172.21.15.46:6789/0 conn(0x7fe6ec140310 legacy=0x7fe6ec06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.426 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.425+0000 7fe6f0c56640 1 -- 172.21.15.46:0/1823010011 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6ec142de0 con 0x7fe6ec06abe0 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.426+0000 7fe6f0c56640 1 --2- 172.21.15.46:0/1823010011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec06abe0 0x7fe6ec1618e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0097d0 tx=0x7fe6dc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a205a9b1bb9267f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.426+0000 7fe6e3fff640 1 -- 172.21.15.46:0/1823010011 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc031800 con 0x7fe6ec06abe0 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.426+0000 7fe6e3fff640 1 -- 172.21.15.46:0/1823010011 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc031960 con 0x7fe6ec06abe0 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.426+0000 7fe6e3fff640 1 -- 172.21.15.46:0/1823010011 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc031cd0 con 0x7fe6ec06abe0 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 -- 172.21.15.46:0/1823010011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec06abe0 msgr2=0x7fe6ec1618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/1823010011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec06abe0 0x7fe6ec1618e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0097d0 tx=0x7fe6dc02b8a0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 -- 172.21.15.46:0/1823010011 shutdown_connections 2023-12-16T21:15:41.427 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/1823010011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec06abe0 0x7fe6ec1618e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 -- 172.21.15.46:0/1823010011 >> 172.21.15.46:0/1823010011 conn(0x7fe6ec0b1e50 msgr2=0x7fe6ec0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 -- 172.21.15.46:0/1823010011 shutdown_connections 2023-12-16T21:15:41.428 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.427+0000 7fe6f36bb640 1 -- 172.21.15.46:0/1823010011 wait complete. 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.428+0000 7fe6f36bb640 1 Processor -- start 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.428+0000 7fe6f36bb640 1 -- start start 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f36bb640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec06abe0 0x7fe6ec15e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f36bb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 0x7fe6ec143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f36bb640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec1409d0 0x7fe6ec144920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f36bb640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec1551a0 con 0x7fe6ec140310 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f36bb640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec154990 con 0x7fe6ec06abe0 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f36bb640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec15ed90 con 0x7fe6ec1409d0 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 0x7fe6ec143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 0x7fe6ec143f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37180/0 (socket says 172.21.15.46:37180) 2023-12-16T21:15:41.430 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 -- 172.21.15.46:0/462171609 learned_addr learned my addr 172.21.15.46:0/462171609 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f1457640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec06abe0 0x7fe6ec15e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f1c58640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec1409d0 0x7fe6ec144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 -- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec1409d0 msgr2=0x7fe6ec144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec1409d0 0x7fe6ec144920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 -- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec06abe0 msgr2=0x7fe6ec15e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec06abe0 0x7fe6ec15e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 -- 172.21.15.46:0/462171609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc009400 con 0x7fe6ec140310 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f1457640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec06abe0 0x7fe6ec15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.429+0000 7fe6f0c56640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 0x7fe6ec143f90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc000c00 tx=0x7fe6dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.430+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc03ac60 con 0x7fe6ec140310 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.430+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc03adc0 con 0x7fe6ec140310 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.430+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ec154be0 con 0x7fe6ec140310 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.430+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc044630 con 0x7fe6ec140310 2023-12-16T21:15:41.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.430+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ec1568a0 con 0x7fe6ec140310 2023-12-16T21:15:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.431+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ec161030 con 0x7fe6ec140310 2023-12-16T21:15:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.431+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe6dc04d450 con 0x7fe6ec140310 2023-12-16T21:15:41.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.432+0000 7fe6e1ffb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe6d006ae40 0x7fe6d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.432+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe6dc02e070 con 0x7fe6ec140310 2023-12-16T21:15:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.432+0000 7fe6f1457640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe6d006ae40 0x7fe6d006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.432+0000 7fe6f1457640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe6d006ae40 0x7fe6d006d300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8007fd0 tx=0x7fe6e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.436+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe6dc076ac0 con 0x7fe6ec140310 2023-12-16T21:15:41.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.613+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6ec069400 con 0x7fe6ec140310 2023-12-16T21:15:41.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.614+0000 7fe6e1ffb640 1 -- 172.21.15.46:0/462171609 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6dc035020 con 0x7fe6ec140310 2023-12-16T21:15:41.614 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe6d006ae40 msgr2=0x7fe6d006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe6d006ae40 0x7fe6d006d300 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8007fd0 tx=0x7fe6e8009040 comp rx=0 tx=0).stop 2023-12-16T21:15:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 msgr2=0x7fe6ec143f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 0x7fe6ec143f90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc000c00 tx=0x7fe6dc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 shutdown_connections 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe6d006ae40 0x7fe6d006d300 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6ec140310 0x7fe6ec143f90 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6ec1409d0 0x7fe6ec144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 --2- 172.21.15.46:0/462171609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6ec06abe0 0x7fe6ec15e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 >> 172.21.15.46:0/462171609 conn(0x7fe6ec0b1e50 msgr2=0x7fe6ec06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 shutdown_connections 2023-12-16T21:15:41.618 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.616+0000 7fe6f36bb640 1 -- 172.21.15.46:0/462171609 wait complete. 2023-12-16T21:15:41.626 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:41.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.737+0000 7f01f4a83640 1 Processor -- start 2023-12-16T21:15:41.736 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.737+0000 7f01f4a83640 1 -- start start 2023-12-16T21:15:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.737+0000 7f01f4a83640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01f015d330 0x7f01f00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.737+0000 7f01f4a83640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f00b2b10 con 0x7f01f00d2910 2023-12-16T21:15:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.737+0000 7f01f4a83640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f00d1590 con 0x7f01f00d3900 2023-12-16T21:15:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.737+0000 7f01f4a83640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f00d1870 con 0x7f01f015d330 2023-12-16T21:15:41.738 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01ee59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01f015d330 0x7f01f00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01ee59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f01f015d330 0x7f01f00b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37196/0 (socket says 172.21.15.46:37196) 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01e7fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f01f00d2910 0x7f01f00d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56428/0 (socket says 172.21.15.46:56428) 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01ee59c640 1 -- 172.21.15.46:0/1963441169 learned_addr learned my addr 172.21.15.46:0/1963441169 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01ee59c640 1 -- 172.21.15.46:0/1963441169 >> v1:172.21.15.121:6789/0 conn(0x7f01f00d3900 legacy=0x7f01f00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01ee59c640 1 -- 172.21.15.46:0/1963441169 >> v1:172.21.15.46:6789/0 conn(0x7f01f00d2910 legacy=0x7f01f00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.738+0000 7f01ee59c640 1 -- 172.21.15.46:0/1963441169 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f00d1b50 con 0x7f01f015d330 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01ee59c640 1 --2- 172.21.15.46:0/1963441169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d330 0x7f01f00b8b80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009870 tx=0x7f01e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f9dad68137586c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01edd9b640 1 -- 172.21.15.46:0/1963441169 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0031870 con 0x7f01f015d330 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01edd9b640 1 -- 172.21.15.46:0/1963441169 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e00319d0 con 0x7f01f015d330 2023-12-16T21:15:41.739 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01edd9b640 1 -- 172.21.15.46:0/1963441169 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e0031d60 con 0x7f01f015d330 2023-12-16T21:15:41.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 -- 172.21.15.46:0/1963441169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d330 msgr2=0x7f01f00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1963441169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d330 0x7f01f00b8b80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009870 tx=0x7f01e002b910 comp rx=0 tx=0).stop 2023-12-16T21:15:41.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 -- 172.21.15.46:0/1963441169 shutdown_connections 2023-12-16T21:15:41.740 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1963441169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f015d330 0x7f01f00b8b80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 -- 172.21.15.46:0/1963441169 >> 172.21.15.46:0/1963441169 conn(0x7f01f0072020 msgr2=0x7f01f0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 -- 172.21.15.46:0/1963441169 shutdown_connections 2023-12-16T21:15:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.739+0000 7f01f4a83640 1 -- 172.21.15.46:0/1963441169 wait complete. 2023-12-16T21:15:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 Processor -- start 2023-12-16T21:15:41.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 -- start start 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00d2910 0x7f01f00ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 0x7f01f00ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f015d330 0x7f01f0152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00caca0 con 0x7f01f00d3900 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00caf80 con 0x7f01f015d330 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01f4a83640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00cb260 con 0x7f01f00d2910 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 0x7f01f00ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 0x7f01f00ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37212/0 (socket says 172.21.15.46:37212) 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.741+0000 7f01e7fff640 1 -- 172.21.15.46:0/1769527452 learned_addr learned my addr 172.21.15.46:0/1769527452 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01ee59c640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00d2910 0x7f01f00ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.743 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01eed9d640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f015d330 0x7f01f0152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e7fff640 1 -- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00d2910 msgr2=0x7f01f00ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e7fff640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00d2910 0x7f01f00ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e7fff640 1 -- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f015d330 msgr2=0x7f01f0152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e7fff640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f015d330 0x7f01f0152250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e7fff640 1 -- 172.21.15.46:0/1769527452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e00094d0 con 0x7f01f00d3900 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e7fff640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 0x7f01f00ca330 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c4d0 tx=0x7f01d800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d8007480 con 0x7f01f00d3900 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d80075e0 con 0x7f01f00d3900 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f00cb540 con 0x7f01f00d3900 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d80049e0 con 0x7f01f00d3900 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.742+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f00cd0c0 con 0x7f01f00d3900 2023-12-16T21:15:41.744 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.743+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f00cba60 con 0x7f01f00d3900 2023-12-16T21:15:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.744+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f01d8004b40 con 0x7f01f00d3900 2023-12-16T21:15:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.744+0000 7f01e6ffd640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d006ae40 0x7f01d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.744+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01d8088610 con 0x7f01f00d3900 2023-12-16T21:15:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.744+0000 7f01ee59c640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d006ae40 0x7f01d006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:41.746 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.745+0000 7f01ee59c640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d006ae40 0x7f01d006d300 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f01e0036000 tx=0x7f01e003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:41.748 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.748+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f01f00cba60 con 0x7f01f00d3900 2023-12-16T21:15:41.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.923+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f01f00cba60 con 0x7f01f00d3900 2023-12-16T21:15:41.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.924+0000 7f01e6ffd640 1 -- 172.21.15.46:0/1769527452 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f01f00cba60 con 0x7f01f00d3900 2023-12-16T21:15:41.924 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d006ae40 msgr2=0x7f01d006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d006ae40 0x7f01d006d300 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f01e0036000 tx=0x7f01e003c040 comp rx=0 tx=0).stop 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 msgr2=0x7f01f00ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 0x7f01f00ca330 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f01d800c4d0 tx=0x7f01d800c990 comp rx=0 tx=0).stop 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 shutdown_connections 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f01d006ae40 0x7f01d006d300 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01f00d3900 0x7f01f00ca330 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01f015d330 0x7f01f0152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 --2- 172.21.15.46:0/1769527452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01f00d2910 0x7f01f00ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 >> 172.21.15.46:0/1769527452 conn(0x7f01f0072020 msgr2=0x7f01f0072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 shutdown_connections 2023-12-16T21:15:41.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:41.926+0000 7f01f4a83640 1 -- 172.21.15.46:0/1769527452 wait complete. 2023-12-16T21:15:41.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:42.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.046+0000 7f714e22d640 1 Processor -- start 2023-12-16T21:15:42.046 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.046+0000 7f714e22d640 1 -- start start 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f714e22d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f714815b5e0 0x7f714815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f714e22d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f714806a4d0 0x7f714806a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f714e22d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7148155b70 con 0x7f7148155790 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f714e22d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714806ee30 con 0x7f714815b5e0 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f714e22d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714806f3b0 con 0x7f714806a4d0 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f7146ffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f714815b5e0 0x7f714815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f71477fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f7148155790 0x7f714806f840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56442/0 (socket says 172.21.15.46:56442) 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f71477fe640 1 -- 172.21.15.46:0/479042774 learned_addr learned my addr 172.21.15.46:0/479042774 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f7147fff640 1 --2- 172.21.15.46:0/479042774 >> v2:172.21.15.121:3300/0 conn(0x7f714806a4d0 0x7f714806a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f7146ffd640 1 -- 172.21.15.46:0/479042774 >> v2:172.21.15.121:3300/0 conn(0x7f714806a4d0 msgr2=0x7f714806a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f7146ffd640 1 --2- 172.21.15.46:0/479042774 >> v2:172.21.15.121:3300/0 conn(0x7f714806a4d0 0x7f714806a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f7146ffd640 1 -- 172.21.15.46:0/479042774 >> v1:172.21.15.46:6789/0 conn(0x7f7148155790 legacy=0x7f714806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f71467fc640 1 -- 172.21.15.46:0/479042774 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625981316 0 0) 0x7f7148155b70 con 0x7f7148155790 2023-12-16T21:15:42.048 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.047+0000 7f7146ffd640 1 -- 172.21.15.46:0/479042774 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714806adf0 con 0x7f714815b5e0 2023-12-16T21:15:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.048+0000 7f7146ffd640 1 --2- 172.21.15.46:0/479042774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f714815ba90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f714806b0e0 tx=0x7f713002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cb659da3cb9f04 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.048+0000 7f71467fc640 1 -- 172.21.15.46:0/479042774 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130031830 con 0x7f714815b5e0 2023-12-16T21:15:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.048+0000 7f71467fc640 1 -- 172.21.15.46:0/479042774 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7130031990 con 0x7f714815b5e0 2023-12-16T21:15:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.048+0000 7f71467fc640 1 -- 172.21.15.46:0/479042774 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7130031d20 con 0x7f714815b5e0 2023-12-16T21:15:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 -- 172.21.15.46:0/479042774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 msgr2=0x7f714815ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.049 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 --2- 172.21.15.46:0/479042774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f714815ba90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f714806b0e0 tx=0x7f713002b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 -- 172.21.15.46:0/479042774 shutdown_connections 2023-12-16T21:15:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 --2- 172.21.15.46:0/479042774 >> v2:172.21.15.121:3300/0 conn(0x7f714806a4d0 0x7f714806a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 --2- 172.21.15.46:0/479042774 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f714815ba90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 -- 172.21.15.46:0/479042774 >> 172.21.15.46:0/479042774 conn(0x7f71480b1e50 msgr2=0x7f71480b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 -- 172.21.15.46:0/479042774 shutdown_connections 2023-12-16T21:15:42.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.049+0000 7f714e22d640 1 -- 172.21.15.46:0/479042774 wait complete. 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.050+0000 7f714e22d640 1 Processor -- start 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.050+0000 7f714e22d640 1 -- start start 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f714e22d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714806a4d0 0x7f7148144880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f714e22d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7148155790 0x7f71481e5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f714e22d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f71481e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f714e22d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71481f2230 con 0x7f714815b5e0 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f714e22d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71481f15b0 con 0x7f7148155790 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f714e22d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71481f1890 con 0x7f714806a4d0 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f71481e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f71481e57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37230/0 (socket says 172.21.15.46:37230) 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 -- 172.21.15.46:0/1379556256 learned_addr learned my addr 172.21.15.46:0/1379556256 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 -- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714806a4d0 msgr2=0x7f7148144880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.052 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714806a4d0 0x7f7148144880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 -- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7148155790 msgr2=0x7f71481e5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7148155790 0x7f71481e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.051+0000 7f7147fff640 1 -- 172.21.15.46:0/1379556256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009400 con 0x7f714815b5e0 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.052+0000 7f7147fff640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f71481e57d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7140007e60 tx=0x7f714000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.052+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140014420 con 0x7f714815b5e0 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.052+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71400145a0 con 0x7f714815b5e0 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.052+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71481f1b70 con 0x7f714815b5e0 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.052+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140014950 con 0x7f714815b5e0 2023-12-16T21:15:42.053 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.052+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7148144dc0 con 0x7f714815b5e0 2023-12-16T21:15:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.053+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71481f2370 con 0x7f714815b5e0 2023-12-16T21:15:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.053+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7140014ab0 con 0x7f714815b5e0 2023-12-16T21:15:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.054+0000 7f713ffff640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f712806ae40 0x7f712806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.054+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7140089f70 con 0x7f714815b5e0 2023-12-16T21:15:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.054+0000 7f71477fe640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f712806ae40 0x7f712806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.054+0000 7f71477fe640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f712806ae40 0x7f712806d300 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7138007fd0 tx=0x7f7138007920 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.057 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.058+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f71400bcea0 con 0x7f714815b5e0 2023-12-16T21:15:42.232 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.233+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7148069400 con 0x7f714815b5e0 2023-12-16T21:15:42.233 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.233+0000 7f713ffff640 1 -- 172.21.15.46:0/1379556256 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7140055020 con 0x7f714815b5e0 2023-12-16T21:15:42.233 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f712806ae40 msgr2=0x7f712806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f712806ae40 0x7f712806d300 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7138007fd0 tx=0x7f7138007920 comp rx=0 tx=0).stop 2023-12-16T21:15:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 msgr2=0x7f71481e57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f71481e57d0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7140007e60 tx=0x7f714000e4b0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.235 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 shutdown_connections 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f712806ae40 0x7f712806d300 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f714815b5e0 0x7f71481e57d0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7148155790 0x7f71481e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 --2- 172.21.15.46:0/1379556256 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714806a4d0 0x7f7148144880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 >> 172.21.15.46:0/1379556256 conn(0x7f71480b1e50 msgr2=0x7f714815d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 shutdown_connections 2023-12-16T21:15:42.236 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.235+0000 7f714e22d640 1 -- 172.21.15.46:0/1379556256 wait complete. 2023-12-16T21:15:42.246 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.353+0000 7f90f5cb4640 1 Processor -- start 2023-12-16T21:15:42.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.353+0000 7f90f5cb4640 1 -- start start 2023-12-16T21:15:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90f5cb4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90f5cb4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f0160fb0 0x7f90f0161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90f5cb4640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f90f00b3ff0 0x7f90f00b4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90f5cb4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f0161d20 con 0x7f90f0160fb0 2023-12-16T21:15:42.357 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90f5cb4640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f006f370 con 0x7f90f00b3ff0 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90f5cb4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f015e760 con 0x7f90f006ea50 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90eeffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f0160fb0 0x7f90f0161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90eeffd640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f90f0160fb0 0x7f90f0161410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37240/0 (socket says 172.21.15.46:37240) 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.354+0000 7f90eeffd640 1 -- 172.21.15.46:0/596051699 learned_addr learned my addr 172.21.15.46:0/596051699 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90ef7fe640 1 --2- 172.21.15.46:0/596051699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90effff640 1 --2- 172.21.15.46:0/596051699 >> v2:172.21.15.121:3300/0 conn(0x7f90f00b3ff0 0x7f90f00b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90eeffd640 1 -- 172.21.15.46:0/596051699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 msgr2=0x7f90f006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90eeffd640 1 --2- 172.21.15.46:0/596051699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f006ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90eeffd640 1 -- 172.21.15.46:0/596051699 >> v2:172.21.15.121:3300/0 conn(0x7f90f00b3ff0 msgr2=0x7f90f00b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90eeffd640 1 --2- 172.21.15.46:0/596051699 >> v2:172.21.15.121:3300/0 conn(0x7f90f00b3ff0 0x7f90f00b4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90eeffd640 1 -- 172.21.15.46:0/596051699 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f015d9a0 con 0x7f90f0160fb0 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.355+0000 7f90eeffd640 1 --2- 172.21.15.46:0/596051699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f0160fb0 0x7f90f0161410 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f90f00b66c0 tx=0x7f90e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76b841c91aeea11f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90ee7fc640 1 -- 172.21.15.46:0/596051699 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40317e0 con 0x7f90f0160fb0 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90ee7fc640 1 -- 172.21.15.46:0/596051699 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e4031940 con 0x7f90f0160fb0 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90ee7fc640 1 -- 172.21.15.46:0/596051699 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4031cd0 con 0x7f90f0160fb0 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 -- 172.21.15.46:0/596051699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f0160fb0 msgr2=0x7f90f0161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/596051699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f0160fb0 0x7f90f0161410 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f90f00b66c0 tx=0x7f90e402b880 comp rx=0 tx=0).stop 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 -- 172.21.15.46:0/596051699 shutdown_connections 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/596051699 >> v2:172.21.15.121:3300/0 conn(0x7f90f00b3ff0 0x7f90f00b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/596051699 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f0160fb0 0x7f90f0161410 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/596051699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 -- 172.21.15.46:0/596051699 >> 172.21.15.46:0/596051699 conn(0x7f90f006a6b0 msgr2=0x7f90f006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 -- 172.21.15.46:0/596051699 shutdown_connections 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.356+0000 7f90f5cb4640 1 -- 172.21.15.46:0/596051699 wait complete. 2023-12-16T21:15:42.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.357+0000 7f90f5cb4640 1 Processor -- start 2023-12-16T21:15:42.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.357+0000 7f90f5cb4640 1 -- start start 2023-12-16T21:15:42.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90f5cb4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f01f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90f5cb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 0x7f90f013da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90f5cb4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f0160fb0 0x7f90f013b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90f5cb4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f006b680 con 0x7f90f00b3ff0 2023-12-16T21:15:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90f5cb4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f0159840 con 0x7f90f0160fb0 2023-12-16T21:15:42.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90f5cb4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f013e0d0 con 0x7f90f006ea50 2023-12-16T21:15:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 0x7f90f013da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 0x7f90f013da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37246/0 (socket says 172.21.15.46:37246) 2023-12-16T21:15:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.358+0000 7f90eeffd640 1 -- 172.21.15.46:0/2673451314 learned_addr learned my addr 172.21.15.46:0/2673451314 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90effff640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f0160fb0 0x7f90f013b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90ef7fe640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f01f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90eeffd640 1 -- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 msgr2=0x7f90f01f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90eeffd640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f01f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90eeffd640 1 -- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f0160fb0 msgr2=0x7f90f013b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90eeffd640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f0160fb0 0x7f90f013b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90eeffd640 1 -- 172.21.15.46:0/2673451314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40094d0 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90eeffd640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 0x7f90f013da10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f90e40090b0 tx=0x7f90e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4032980 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e4032ae0 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e4032e90 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f006b920 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.359+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f013e3b0 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.360+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f006be40 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.360+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f90e404d3f0 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.361+0000 7f90d7fff640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90dc06ae40 0x7f90dc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.361+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90e40ab5a0 con 0x7f90f00b3ff0 2023-12-16T21:15:42.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.361+0000 7f90ef7fe640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90dc06ae40 0x7f90dc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.361+0000 7f90ef7fe640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90dc06ae40 0x7f90dc06d300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f90e0006fd0 tx=0x7f90e0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.365+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f90f006be40 con 0x7f90f00b3ff0 2023-12-16T21:15:42.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.553+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90f006be40 con 0x7f90f00b3ff0 2023-12-16T21:15:42.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.554+0000 7f90d7fff640 1 -- 172.21.15.46:0/2673451314 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7f90f006be40 con 0x7f90f00b3ff0 2023-12-16T21:15:42.553 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:42.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90dc06ae40 msgr2=0x7f90dc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90dc06ae40 0x7f90dc06d300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f90e0006fd0 tx=0x7f90e0008040 comp rx=0 tx=0).stop 2023-12-16T21:15:42.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 msgr2=0x7f90f013da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 0x7f90f013da10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f90e40090b0 tx=0x7f90e40046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 shutdown_connections 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f90dc06ae40 0x7f90dc06d300 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f90f00b3ff0 0x7f90f013da10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f0160fb0 0x7f90f013b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 --2- 172.21.15.46:0/2673451314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f006ea50 0x7f90f01f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.555+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 >> 172.21.15.46:0/2673451314 conn(0x7f90f006a6b0 msgr2=0x7f90f006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.556+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 shutdown_connections 2023-12-16T21:15:42.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.556+0000 7f90f5cb4640 1 -- 172.21.15.46:0/2673451314 wait complete. 2023-12-16T21:15:42.566 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:15:42.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:42.566 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:42.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[109562]: fuse finished with error 0 and tester_r 0 2023-12-16T21:15:42.597 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:42.636 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:42.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:42.636 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:15:42.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:15:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.839+0000 7f691678a640 1 Processor -- start 2023-12-16T21:15:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.839+0000 7f691678a640 1 -- start start 2023-12-16T21:15:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f691678a640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6910150560 0x7f69100b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f691678a640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69100b49f0 con 0x7f6910150940 2023-12-16T21:15:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f691678a640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69100b3de0 con 0x7f69100b3390 2023-12-16T21:15:42.843 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f691678a640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69100b40c0 con 0x7f6910150560 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f6914d27640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f6910150940 0x7f69100ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56448/0 (socket says 172.21.15.46:56448) 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f6914d27640 1 -- 172.21.15.46:0/1283381825 learned_addr learned my addr 172.21.15.46:0/1283381825 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690ffff640 1 --2- 172.21.15.46:0/1283381825 >> v2:172.21.15.121:3300/0 conn(0x7f6910150560 0x7f69100b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136607520 0 0) 0x7f69100b49f0 con 0x7f6910150940 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f8003520 con 0x7f6910150940 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991704414 0 0) 0x7f68f8003520 con 0x7f6910150940 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 >> v2:172.21.15.121:3300/0 conn(0x7f6910150560 msgr2=0x7f69100b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 --2- 172.21.15.46:0/1283381825 >> v2:172.21.15.121:3300/0 conn(0x7f6910150560 0x7f69100b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 >> v1:172.21.15.121:6789/0 conn(0x7f69100b3390 legacy=0x7f691015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69100b43a0 con 0x7f6910150940 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f69080019f0 con 0x7f6910150940 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 >> v1:172.21.15.46:6789/0 conn(0x7f6910150940 legacy=0x7f69100ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.840+0000 7f690effd640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f8003970 0x7f68f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690effd640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f80047c0 0x7f68f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690effd640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f68f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69100b40c0 con 0x7f68f8005180 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68f800d130 con 0x7f68f80047c0 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68f800d2f0 con 0x7f68f8003970 2023-12-16T21:15:42.845 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f6914d27640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f68f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690f7fe640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f80047c0 0x7f68f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f690ffff640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f8003970 0x7f68f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f6914d27640 1 -- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f8003970 msgr2=0x7f68f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f6914d27640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68f8003970 0x7f68f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f6914d27640 1 -- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f80047c0 msgr2=0x7f68f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f6914d27640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68f80047c0 0x7f68f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.841+0000 7f6914d27640 1 -- 172.21.15.46:0/1283381825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68f8003520 con 0x7f68f8005180 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f6914d27640 1 -- 172.21.15.46:0/1283381825 reap_dead start 2023-12-16T21:15:42.846 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f6914d27640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f68f800b980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f690800a830 tx=0x7f68f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdeea6a88664f57 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69080072d0 con 0x7f68f8005180 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690802f070 con 0x7f68f8005180 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f690effd640 1 -- 172.21.15.46:0/1283381825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69080076b0 con 0x7f68f8005180 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 -- 172.21.15.46:0/1283381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 msgr2=0x7f68f800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f68f800b980 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f690800a830 tx=0x7f68f80057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 -- 172.21.15.46:0/1283381825 shutdown_connections 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 --2- 172.21.15.46:0/1283381825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f68f800b980 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 -- 172.21.15.46:0/1283381825 >> 172.21.15.46:0/1283381825 conn(0x7f691006a4d0 msgr2=0x7f691006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 -- 172.21.15.46:0/1283381825 shutdown_connections 2023-12-16T21:15:42.847 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.842+0000 7f691678a640 1 -- 172.21.15.46:0/1283381825 wait complete. 2023-12-16T21:15:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 Processor -- start 2023-12-16T21:15:42.848 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 -- start start 2023-12-16T21:15:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f6910146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f691014fb40 0x7f69100ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69100705c0 0x7f6910070a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910070fb0 con 0x7f68f8005180 2023-12-16T21:15:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6910158900 con 0x7f69100705c0 2023-12-16T21:15:42.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.844+0000 7f691678a640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69100ad250 con 0x7f691014fb40 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f6910146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f6910146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37258/0 (socket says 172.21.15.46:37258) 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 -- 172.21.15.46:0/1717772687 learned_addr learned my addr 172.21.15.46:0/1717772687 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f6914d27640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69100705c0 0x7f6910070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690f7fe640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f691014fb40 0x7f69100ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 -- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f691014fb40 msgr2=0x7f69100ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f691014fb40 0x7f69100ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 -- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69100705c0 msgr2=0x7f6910070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:42.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69100705c0 0x7f6910070a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 -- 172.21.15.46:0/1717772687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69100ad530 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f6914d27640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69100705c0 0x7f6910070a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690ffff640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f6910146ba0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f69000118e0 tx=0x7f6900011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690000faf0 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690000fc50 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69100ad810 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900014440 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.845+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69100ace20 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.846+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6910069400 con 0x7f68f8005180 2023-12-16T21:15:42.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.847+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f69000145a0 con 0x7f68f8005180 2023-12-16T21:15:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.847+0000 7f690cff9640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68fc06ae60 0x7f68fc06d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.847+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6900089ea0 con 0x7f68f8005180 2023-12-16T21:15:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.847+0000 7f690f7fe640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68fc06ae60 0x7f68fc06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.848+0000 7f690f7fe640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68fc06ae60 0x7f68fc06d320 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f69101577b0 tx=0x7f6904009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:42.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:42.852+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6900054f50 con 0x7f68f8005180 2023-12-16T21:15:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.065+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f691015d360 con 0x7f68f8005180 2023-12-16T21:15:43.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.065+0000 7f690cff9640 1 -- 172.21.15.46:0/1717772687 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6900058570 con 0x7f68f8005180 2023-12-16T21:15:43.065 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:15:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68fc06ae60 msgr2=0x7f68fc06d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68fc06ae60 0x7f68fc06d320 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f69101577b0 tx=0x7f6904009040 comp rx=0 tx=0).stop 2023-12-16T21:15:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 msgr2=0x7f6910146ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:43.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f6910146ba0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f69000118e0 tx=0x7f6900011da0 comp rx=0 tx=0).stop 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 shutdown_connections 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f68fc06ae60 0x7f68fc06d320 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69100705c0 0x7f6910070a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f691014fb40 0x7f69100ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 --2- 172.21.15.46:0/1717772687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68f8005180 0x7f6910146ba0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 >> 172.21.15.46:0/1717772687 conn(0x7f691006a4d0 msgr2=0x7f69100ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 shutdown_connections 2023-12-16T21:15:43.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43.067+0000 7f691678a640 1 -- 172.21.15.46:0/1717772687 wait complete. 2023-12-16T21:15:43.077 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:15:43.077 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:15:43.078 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:43.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.078 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:15:43.094 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:15:43.094 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:15:43.094 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:15:43.094 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42542sec preferred_lft 42542sec 2023-12-16T21:15:43.095 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:43.096 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:43.097 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::20ee:6cff:fee1:5615/64 scope link 2023-12-16T21:15:43.097 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:43.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.097 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:15:43.164 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:15:43.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.166 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:15:43.221 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:15:43.221 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:15:43.222 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:15:43.222 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:15:43.222 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:15:43.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.222 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:15:43.222 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:15:43.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:15:43.222 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:15:43.223 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:15:43.223 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:15:43.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.327 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:15:43.342 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:15:43.342 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:15:43.343 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:43.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:15:43.413 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:15:43.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:43.413 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:43.469 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:43.469 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:43.469 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:43.524 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:43.595 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:43.651 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:43.652 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:43.653 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:43.654 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:43.654 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:43.654 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:43.654 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:43.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.654 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:43.707 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:15:43.707 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:15:43.750 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:43.781 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:43.838 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:43.838 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:43.838 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:43.838 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:43.838 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:43.839 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:43.840 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:43.841 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:43.842 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:43.842 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:43.842 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:43.842 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:43.842 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:43.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.843 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:43.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:15:43.844+0000 7f5b0a5f4480 -1 init, newargv = 0x56158d942840 newargc=15 2023-12-16T21:15:43.844 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[110525]: starting ceph client 2023-12-16T21:15:43.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[110525]: starting fuse 2023-12-16T21:15:43.863 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:15:43.863 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:15:43.863 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:15:43.863 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:15:43.863 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:15:43.863 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:15:43.863 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:15:43.863 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:15:43.864 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:15:43.865 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:15:43.865 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:43.865 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:15:43.865 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:15:43.988 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.110525.asok 2023-12-16T21:15:43.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:15:43.999 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.110525.asok 2023-12-16T21:15:43.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:43.999 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.110525.asok status 2023-12-16T21:15:44.141 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "110525", 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6168, 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:15:44.142 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6168 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4016975360 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4016975360 2023-12-16T21:15:44.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6168 192.168.0.1:0/4016975360", 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/4016975360", 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 20, 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 50, 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:15:44.144 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:15:44.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:44.150 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:15:44.150 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:44.165 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:15:44.166 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:44.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:44.166 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:44.237 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2023-12-16T21:15:44.237 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:15:44.237 DEBUG:teuthology.orchestra.run.smithi046:> import errno 2023-12-16T21:15:44.237 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> try: 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> if False: 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> ret = os.read(fd, 1024) 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> else: 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> os.write(fd, b'"'"'content'"'"') 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> if e.errno != errno.EPERM: 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> raise 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> else: 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("client does not check permission of data pool") 2023-12-16T21:15:44.238 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:15:44.350 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:15:44.357 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2023-12-16T21:15:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.510+0000 7fcc9f3e5640 1 Processor -- start 2023-12-16T21:15:44.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.510+0000 7fcc9f3e5640 1 -- start start 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9f3e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc98146d70 0x7fcc980b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9f3e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc980b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9f3e5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc980cf660 con 0x7fcc98146010 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9f3e5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc98161170 con 0x7fcc98146d70 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9f3e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc98161450 con 0x7fcc980b4060 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9c980640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fcc98146d70 0x7fcc980b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.513 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9d181640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fcc98146010 0x7fcc980cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56458/0 (socket says 172.21.15.46:56458) 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9d181640 1 -- 172.21.15.46:0/2332471878 learned_addr learned my addr 172.21.15.46:0/2332471878 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9d982640 1 --2- 172.21.15.46:0/2332471878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc980b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9c980640 1 -- 172.21.15.46:0/2332471878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 msgr2=0x7fcc980b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9c980640 1 --2- 172.21.15.46:0/2332471878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc980b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9c980640 1 -- 172.21.15.46:0/2332471878 >> v1:172.21.15.46:6789/0 conn(0x7fcc98146010 legacy=0x7fcc980cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc9c980640 1 -- 172.21.15.46:0/2332471878 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc98161730 con 0x7fcc98146d70 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.511+0000 7fcc97fff640 1 -- 172.21.15.46:0/2332471878 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657564965 0 0) 0x7fcc980cf660 con 0x7fcc98146010 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.512+0000 7fcc9c980640 1 --2- 172.21.15.46:0/2332471878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc980b2cf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c0097a0 tx=0x7fcc8c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227c5aa39a54846f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.512+0000 7fcc97fff640 1 -- 172.21.15.46:0/2332471878 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c031830 con 0x7fcc98146d70 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.512+0000 7fcc97fff640 1 -- 172.21.15.46:0/2332471878 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8c031990 con 0x7fcc98146d70 2023-12-16T21:15:44.514 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.512+0000 7fcc97fff640 1 -- 172.21.15.46:0/2332471878 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c031d20 con 0x7fcc98146d70 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/2332471878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 msgr2=0x7fcc980b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/2332471878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc980b2cf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c0097a0 tx=0x7fcc8c02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/2332471878 shutdown_connections 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/2332471878 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc980b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/2332471878 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc980b2cf0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/2332471878 >> 172.21.15.46:0/2332471878 conn(0x7fcc9806a4d0 msgr2=0x7fcc9806a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/2332471878 shutdown_connections 2023-12-16T21:15:44.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.513+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/2332471878 wait complete. 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.514+0000 7fcc9f3e5640 1 Processor -- start 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.514+0000 7fcc9f3e5640 1 -- start start 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9f3e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc9813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9f3e5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc98146010 0x7fcc9813efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9f3e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc9815c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9f3e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc981409f0 con 0x7fcc98146d70 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9f3e5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9813f510 con 0x7fcc98146010 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9f3e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9813fde0 con 0x7fcc980b4060 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc9815c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc9815c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37290/0 (socket says 172.21.15.46:37290) 2023-12-16T21:15:44.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 -- 172.21.15.46:0/1283780720 learned_addr learned my addr 172.21.15.46:0/1283780720 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d181640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc9813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9c980640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc98146010 0x7fcc9813efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 -- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 msgr2=0x7fcc9813f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc9813f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 -- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc98146010 msgr2=0x7fcc9813efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc98146010 0x7fcc9813efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.515+0000 7fcc9d982640 1 -- 172.21.15.46:0/1283780720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8c009400 con 0x7fcc98146d70 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc9c980640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc98146010 0x7fcc9813efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc9d982640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc9815c760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcc88007e40 tx=0x7fcc8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc88014620 con 0x7fcc98146d70 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc8800ee90 con 0x7fcc98146d70 2023-12-16T21:15:44.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc98146570 con 0x7fcc98146d70 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc880149e0 con 0x7fcc98146d70 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.516+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc981403a0 con 0x7fcc98146d70 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.517+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc98140b60 con 0x7fcc98146d70 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.517+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fcc88014b40 con 0x7fcc98146d70 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.518+0000 7fcc95ffb640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcc8406ae40 0x7fcc8406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.518+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc88089f30 con 0x7fcc98146d70 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.518+0000 7fcc9d181640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcc8406ae40 0x7fcc8406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.519+0000 7fcc9d181640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcc8406ae40 0x7fcc8406d300 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcc90000f60 tx=0x7fcc90007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:44.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.522+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fcc88054fe0 con 0x7fcc98146d70 2023-12-16T21:15:44.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.767+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) v1 -- 0x7fcc98069400 con 0x7fcc98146d70 2023-12-16T21:15:44.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.769+0000 7fcc95ffb640 1 -- 172.21.15.46:0/1283780720 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v21) v1 ==== 183+0+140 (secure 0 0 0) 0x7fcc88058600 con 0x7fcc98146d70 2023-12-16T21:15:44.768 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:15:44.769 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:15:44.769 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2023-12-16T21:15:44.769 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:15:44.769 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow w pool=cephfs_data" 2023-12-16T21:15:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:15:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcc8406ae40 msgr2=0x7fcc8406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcc8406ae40 0x7fcc8406d300 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fcc90000f60 tx=0x7fcc90007d50 comp rx=0 tx=0).stop 2023-12-16T21:15:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 msgr2=0x7fcc9815c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc9815c760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fcc88007e40 tx=0x7fcc8800e4b0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 shutdown_connections 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcc8406ae40 0x7fcc8406d300 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc98146d70 0x7fcc9815c760 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc98146010 0x7fcc9813efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.771+0000 7fcc9f3e5640 1 --2- 172.21.15.46:0/1283780720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcc980b4060 0x7fcc9813f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.772+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 >> 172.21.15.46:0/1283780720 conn(0x7fcc9806a4d0 msgr2=0x7fcc9806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.772+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 shutdown_connections 2023-12-16T21:15:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.772+0000 7fcc9f3e5640 1 -- 172.21.15.46:0/1283780720 wait complete. 2023-12-16T21:15:44.782 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:44.782 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:44.798 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:44.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.945+0000 7ff8ba089640 1 Processor -- start 2023-12-16T21:15:44.945 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.945+0000 7ff8ba089640 1 -- start start 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.945+0000 7ff8ba089640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8b4143490 0x7ff8b40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8ba089640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b415e2a0 con 0x7ff8b415e730 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8ba089640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8b415b240 con 0x7ff8b415abf0 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8ba089640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8b415b520 con 0x7ff8b4143490 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8b37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7ff8b4143490 0x7ff8b40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8b2ffd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7ff8b415e730 0x7ff8b415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56468/0 (socket says 172.21.15.46:56468) 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8b2ffd640 1 -- 172.21.15.46:0/319047426 learned_addr learned my addr 172.21.15.46:0/319047426 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8b37fe640 1 -- 172.21.15.46:0/319047426 >> v1:172.21.15.121:6789/0 conn(0x7ff8b415abf0 legacy=0x7ff8b40b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8b37fe640 1 -- 172.21.15.46:0/319047426 >> v1:172.21.15.46:6789/0 conn(0x7ff8b415e730 legacy=0x7ff8b415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.946+0000 7ff8b37fe640 1 -- 172.21.15.46:0/319047426 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b415dc50 con 0x7ff8b4143490 2023-12-16T21:15:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8b37fe640 1 --2- 172.21.15.46:0/319047426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b4143490 0x7ff8b40b0cf0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40097a0 tx=0x7ff8a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2a80a1501f4375 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8b27fc640 1 -- 172.21.15.46:0/319047426 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4031830 con 0x7ff8b4143490 2023-12-16T21:15:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8b27fc640 1 -- 172.21.15.46:0/319047426 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a4031990 con 0x7ff8b4143490 2023-12-16T21:15:44.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8b27fc640 1 -- 172.21.15.46:0/319047426 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a4031d20 con 0x7ff8b4143490 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8ba089640 1 -- 172.21.15.46:0/319047426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b4143490 msgr2=0x7ff8b40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8ba089640 1 --2- 172.21.15.46:0/319047426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b4143490 0x7ff8b40b0cf0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff8a40097a0 tx=0x7ff8a402b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8ba089640 1 -- 172.21.15.46:0/319047426 shutdown_connections 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8ba089640 1 --2- 172.21.15.46:0/319047426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b4143490 0x7ff8b40b0cf0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.947+0000 7ff8ba089640 1 -- 172.21.15.46:0/319047426 >> 172.21.15.46:0/319047426 conn(0x7ff8b406a4d0 msgr2=0x7ff8b406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.948+0000 7ff8ba089640 1 -- 172.21.15.46:0/319047426 shutdown_connections 2023-12-16T21:15:44.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.948+0000 7ff8ba089640 1 -- 172.21.15.46:0/319047426 wait complete. 2023-12-16T21:15:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.949+0000 7ff8ba089640 1 Processor -- start 2023-12-16T21:15:44.949 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.949+0000 7ff8ba089640 1 -- start start 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8b4143490 0x7ff8b40c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8b415abf0 0x7ff8b4070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 0x7ff8b40a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b4132210 con 0x7ff8b415e730 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b41324f0 con 0x7ff8b4143490 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b41327d0 con 0x7ff8b415abf0 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 0x7ff8b40a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 0x7ff8b40a88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37308/0 (socket says 172.21.15.46:37308) 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 -- 172.21.15.46:0/3402889903 learned_addr learned my addr 172.21.15.46:0/3402889903 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b37fe640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8b4143490 0x7ff8b40c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.951 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b2ffd640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8b415abf0 0x7ff8b4070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 -- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8b415abf0 msgr2=0x7ff8b4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8b415abf0 0x7ff8b4070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 -- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8b4143490 msgr2=0x7ff8b40c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8b4143490 0x7ff8b40c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 -- 172.21.15.46:0/3402889903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a4009400 con 0x7ff8b415e730 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b37fe640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8b4143490 0x7ff8b40c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8b3fff640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 0x7ff8b40a88d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00c780 tx=0x7ff8ac00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac004b60 con 0x7ff8b415e730 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ac00ce90 con 0x7ff8b415e730 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b4132ab0 con 0x7ff8b415e730 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00a850 con 0x7ff8b415e730 2023-12-16T21:15:44.952 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.950+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b4070b40 con 0x7ff8b415e730 2023-12-16T21:15:44.953 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.951+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b40b4380 con 0x7ff8b415e730 2023-12-16T21:15:44.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.952+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7ff8ac00a9b0 con 0x7ff8b415e730 2023-12-16T21:15:44.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.952+0000 7ff89bfff640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff8a006ae40 0x7ff8a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:44.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.952+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8ac0897c0 con 0x7ff8b415e730 2023-12-16T21:15:44.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.953+0000 7ff8b37fe640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff8a006ae40 0x7ff8a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:44.954 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.953+0000 7ff8b37fe640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff8a006ae40 0x7ff8a006d300 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4036000 tx=0x7ff8a403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:44.956 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:44.956+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7ff8ac054870 con 0x7ff8b415e730 2023-12-16T21:15:45.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.134+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8b4069400 con 0x7ff8b415e730 2023-12-16T21:15:45.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.135+0000 7ff89bfff640 1 -- 172.21.15.46:0/3402889903 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7ff8ac057e90 con 0x7ff8b415e730 2023-12-16T21:15:45.135 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:45.135 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:45.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff8a006ae40 msgr2=0x7ff8a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff8a006ae40 0x7ff8a006d300 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff8a4036000 tx=0x7ff8a403c040 comp rx=0 tx=0).stop 2023-12-16T21:15:45.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 msgr2=0x7ff8b40a88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 0x7ff8b40a88d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00c780 tx=0x7ff8ac00cc40 comp rx=0 tx=0).stop 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 shutdown_connections 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7ff8a006ae40 0x7ff8a006d300 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8b415e730 0x7ff8b40a88d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8b415abf0 0x7ff8b4070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 --2- 172.21.15.46:0/3402889903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8b4143490 0x7ff8b40c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.137+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 >> 172.21.15.46:0/3402889903 conn(0x7ff8b406a4d0 msgr2=0x7ff8b415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.138+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 shutdown_connections 2023-12-16T21:15:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.138+0000 7ff8ba089640 1 -- 172.21.15.46:0/3402889903 wait complete. 2023-12-16T21:15:45.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:15:29.641456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:15:30.976721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:45.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:45.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.258+0000 7fc07d851640 1 Processor -- start 2023-12-16T21:15:45.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.258+0000 7fc07d851640 1 -- start start 2023-12-16T21:15:45.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.259+0000 7fc07d851640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc07815d330 0x7fc0780b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.259+0000 7fc07d851640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d3900 0x7fc0780b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.259+0000 7fc07d851640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0780b2b10 con 0x7fc0780d2910 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.259+0000 7fc07d851640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0780d1590 con 0x7fc07815d330 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.259+0000 7fc07d851640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0780d1870 con 0x7fc0780d3900 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc076ffd640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc07815d330 0x7fc0780b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc0767fc640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc0780d2910 0x7fc0780d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:32862/0 (socket says 172.21.15.46:32862) 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc0777fe640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d3900 0x7fc0780b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc0767fc640 1 -- 172.21.15.46:0/2452799853 learned_addr learned my addr 172.21.15.46:0/2452799853 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc076ffd640 1 -- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d3900 msgr2=0x7fc0780b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc076ffd640 1 --2- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d3900 0x7fc0780b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc076ffd640 1 -- 172.21.15.46:0/2452799853 >> v1:172.21.15.121:6789/0 conn(0x7fc0780d2910 legacy=0x7fc0780d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc076ffd640 1 -- 172.21.15.46:0/2452799853 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0780d1b50 con 0x7fc07815d330 2023-12-16T21:15:45.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.260+0000 7fc0777fe640 1 --2- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d3900 0x7fc0780b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:45.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc076ffd640 1 --2- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc07815d330 0x7fc0780b8b80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc068009d90 tx=0x7fc06802be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c9c708ad2d2155e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc075ffb640 1 -- 172.21.15.46:0/2452799853 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06802cbc0 con 0x7fc07815d330 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc075ffb640 1 -- 172.21.15.46:0/2452799853 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06802cd20 con 0x7fc07815d330 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc075ffb640 1 -- 172.21.15.46:0/2452799853 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc068034660 con 0x7fc07815d330 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc07d851640 1 -- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc07815d330 msgr2=0x7fc0780b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc07d851640 1 --2- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc07815d330 0x7fc0780b8b80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc068009d90 tx=0x7fc06802be10 comp rx=0 tx=0).stop 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc07d851640 1 -- 172.21.15.46:0/2452799853 shutdown_connections 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc07d851640 1 --2- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d3900 0x7fc0780b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.261+0000 7fc07d851640 1 --2- 172.21.15.46:0/2452799853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc07815d330 0x7fc0780b8b80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.262+0000 7fc07d851640 1 -- 172.21.15.46:0/2452799853 >> 172.21.15.46:0/2452799853 conn(0x7fc078072020 msgr2=0x7fc078072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.262+0000 7fc07d851640 1 -- 172.21.15.46:0/2452799853 shutdown_connections 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.262+0000 7fc07d851640 1 -- 172.21.15.46:0/2452799853 wait complete. 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.263+0000 7fc07d851640 1 Processor -- start 2023-12-16T21:15:45.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.263+0000 7fc07d851640 1 -- start start 2023-12-16T21:15:45.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc07d851640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d2910 0x7fc07815e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc07d851640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0780d3900 0x7fc0780b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc07d851640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 0x7fc07814b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc07d851640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07815d920 con 0x7fc07815d330 2023-12-16T21:15:45.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc07d851640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc07815db70 con 0x7fc0780d3900 2023-12-16T21:15:45.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc07d851640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc07815de50 con 0x7fc0780d2910 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 0x7fc07814b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 0x7fc07814b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37324/0 (socket says 172.21.15.46:37324) 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 -- 172.21.15.46:0/4116206213 learned_addr learned my addr 172.21.15.46:0/4116206213 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0767fc640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0780d3900 0x7fc0780b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc076ffd640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d2910 0x7fc07815e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 -- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d2910 msgr2=0x7fc07815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d2910 0x7fc07815e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 -- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0780d3900 msgr2=0x7fc0780b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0780d3900 0x7fc0780b3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 -- 172.21.15.46:0/4116206213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc070009400 con 0x7fc07815d330 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc0777fe640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 0x7fc07814b2e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc07000e930 tx=0x7fc07000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc070009700 con 0x7fc07815d330 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.264+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07000ca00 con 0x7fc07815d330 2023-12-16T21:15:45.267 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.265+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc068009a70 con 0x7fc07815d330 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.265+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07000cd90 con 0x7fc07815d330 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.265+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc07815e410 con 0x7fc07815d330 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.265+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc078069400 con 0x7fc07815d330 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.266+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc070013470 con 0x7fc07815d330 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.266+0000 7fc06ffff640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc05806ae40 0x7fc05806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.267+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc0700890a0 con 0x7fc07815d330 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.267+0000 7fc076ffd640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc05806ae40 0x7fc05806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.268 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.267+0000 7fc076ffd640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc05806ae40 0x7fc05806d300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc068036000 tx=0x7fc068030000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.270 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.271+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc070054350 con 0x7fc07815d330 2023-12-16T21:15:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.445+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc07815e5d0 con 0x7fc07815d330 2023-12-16T21:15:45.445 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.445+0000 7fc06ffff640 1 -- 172.21.15.46:0/4116206213 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc070057970 con 0x7fc07815d330 2023-12-16T21:15:45.445 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:45.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc05806ae40 msgr2=0x7fc05806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.447 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc05806ae40 0x7fc05806d300 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc068036000 tx=0x7fc068030000 comp rx=0 tx=0).stop 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 msgr2=0x7fc07814b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 0x7fc07814b2e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc07000e930 tx=0x7fc07000edf0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 shutdown_connections 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc05806ae40 0x7fc05806d300 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc07815d330 0x7fc07814b2e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0780d3900 0x7fc0780b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 --2- 172.21.15.46:0/4116206213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0780d2910 0x7fc07815e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.447+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 >> 172.21.15.46:0/4116206213 conn(0x7fc078072020 msgr2=0x7fc078072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.448+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 shutdown_connections 2023-12-16T21:15:45.448 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.448+0000 7fc07d851640 1 -- 172.21.15.46:0/4116206213 wait complete. 2023-12-16T21:15:45.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.567+0000 7f717fdc9640 1 Processor -- start 2023-12-16T21:15:45.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.567+0000 7f717fdc9640 1 -- start start 2023-12-16T21:15:45.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717fdc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f71780b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717fdc9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71780b3390 0x7f717815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717fdc9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71780b49f0 con 0x7f7178150940 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717fdc9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71780b3de0 con 0x7f71780b3390 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717fdc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71780b40c0 con 0x7f7178150560 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717d364640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71780b3390 0x7f717815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717d364640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71780b3390 0x7f717815d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37336/0 (socket says 172.21.15.46:37336) 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.568+0000 7f717d364640 1 -- 172.21.15.46:0/3410156074 learned_addr learned my addr 172.21.15.46:0/3410156074 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.569+0000 7f717db65640 1 --2- 172.21.15.46:0/3410156074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f71780b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.569+0000 7f717d364640 1 -- 172.21.15.46:0/3410156074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 msgr2=0x7f71780b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.569+0000 7f717d364640 1 --2- 172.21.15.46:0/3410156074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f71780b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.569+0000 7f717d364640 1 -- 172.21.15.46:0/3410156074 >> v1:172.21.15.121:6789/0 conn(0x7f7178150940 legacy=0x7f71780ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.569+0000 7f717d364640 1 -- 172.21.15.46:0/3410156074 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71780b43a0 con 0x7f71780b3390 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717d364640 1 --2- 172.21.15.46:0/3410156074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f717815d200 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cac525fd80b80b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717cb63640 1 -- 172.21.15.46:0/3410156074 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168030840 con 0x7f71780b3390 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717cb63640 1 -- 172.21.15.46:0/3410156074 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71680309a0 con 0x7f71780b3390 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717cb63640 1 -- 172.21.15.46:0/3410156074 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168030d30 con 0x7f71780b3390 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 -- 172.21.15.46:0/3410156074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 msgr2=0x7f717815d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 --2- 172.21.15.46:0/3410156074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f717815d200 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f7168009870 tx=0x7f716802b980 comp rx=0 tx=0).stop 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 -- 172.21.15.46:0/3410156074 shutdown_connections 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 --2- 172.21.15.46:0/3410156074 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f717815d200 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 --2- 172.21.15.46:0/3410156074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f71780b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 -- 172.21.15.46:0/3410156074 >> 172.21.15.46:0/3410156074 conn(0x7f717806a4d0 msgr2=0x7f717806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 -- 172.21.15.46:0/3410156074 shutdown_connections 2023-12-16T21:15:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.570+0000 7f717fdc9640 1 -- 172.21.15.46:0/3410156074 wait complete. 2023-12-16T21:15:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 Processor -- start 2023-12-16T21:15:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 -- start start 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f7178140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f717814f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7178144770 0x7f717814fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7178150d30 con 0x7f71780b3390 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71781515d0 con 0x7f7178144770 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717fdc9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71781467d0 con 0x7f7178150560 2023-12-16T21:15:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717db65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f7178140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717db65640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f7178140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37340/0 (socket says 172.21.15.46:37340) 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.572+0000 7f717db65640 1 -- 172.21.15.46:0/4167051195 learned_addr learned my addr 172.21.15.46:0/4167051195 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717d364640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f717814f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717e366640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7178144770 0x7f717814fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717db65640 1 -- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 msgr2=0x7f717814f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717db65640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f717814f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717db65640 1 -- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7178144770 msgr2=0x7f717814fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717db65640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7178144770 0x7f717814fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717db65640 1 -- 172.21.15.46:0/4167051195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71680094d0 con 0x7f71780b3390 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717db65640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f7178140410 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f717400e9b0 tx=0x7f717400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174009670 con 0x7f71780b3390 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717400c9a0 con 0x7f71780b3390 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7178150f80 con 0x7f71780b3390 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717400cd50 con 0x7f71780b3390 2023-12-16T21:15:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.573+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7178146480 con 0x7f71780b3390 2023-12-16T21:15:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.574+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178069400 con 0x7f71780b3390 2023-12-16T21:15:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.575+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7174013470 con 0x7f71780b3390 2023-12-16T21:15:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.575+0000 7f716e7fc640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f715c06ae70 0x7f715c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.575+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71740897a0 con 0x7f71780b3390 2023-12-16T21:15:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.575+0000 7f717d364640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f715c06ae70 0x7f715c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.576+0000 7f717d364640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f715c06ae70 0x7f715c06d330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7168036000 tx=0x7f7168008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.579+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f71740548e0 con 0x7f71780b3390 2023-12-16T21:15:45.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.754+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71780ceb30 con 0x7f71780b3390 2023-12-16T21:15:45.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.754+0000 7f716e7fc640 1 -- 172.21.15.46:0/4167051195 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7174057f00 con 0x7f71780b3390 2023-12-16T21:15:45.754 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f715c06ae70 msgr2=0x7f715c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f715c06ae70 0x7f715c06d330 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7168036000 tx=0x7f7168008030 comp rx=0 tx=0).stop 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 msgr2=0x7f7178140410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f7178140410 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f717400e9b0 tx=0x7f717400ee70 comp rx=0 tx=0).stop 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 shutdown_connections 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f715c06ae70 0x7f715c06d330 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7178144770 0x7f717814fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7178150560 0x7f717814f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 --2- 172.21.15.46:0/4167051195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71780b3390 0x7f7178140410 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 >> 172.21.15.46:0/4167051195 conn(0x7f717806a4d0 msgr2=0x7f71780b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:45.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 shutdown_connections 2023-12-16T21:15:45.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.756+0000 7f717fdc9640 1 -- 172.21.15.46:0/4167051195 wait complete. 2023-12-16T21:15:45.766 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.876+0000 7f16f684f640 1 Processor -- start 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.876+0000 7f16f684f640 1 -- start start 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.877+0000 7f16f684f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f16f00d2910 0x7f16f00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.877+0000 7f16f684f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00b2b10 con 0x7f16f015d330 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.877+0000 7f16f684f640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f00d1590 con 0x7f16f00d3730 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.877+0000 7f16f684f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f00d3bc0 con 0x7f16f00d2910 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.877+0000 7f16effff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f16f015d330 0x7f16f00b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56482/0 (socket says 172.21.15.46:56482) 2023-12-16T21:15:45.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.877+0000 7f16effff640 1 -- 172.21.15.46:0/62630796 learned_addr learned my addr 172.21.15.46:0/62630796 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16ef7fe640 1 --2- 172.21.15.46:0/62630796 >> v2:172.21.15.121:3300/0 conn(0x7f16f00d2910 0x7f16f00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522652956 0 0) 0x7f16f00b2b10 con 0x7f16f015d330 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16d8003520 con 0x7f16f015d330 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157034411 0 0) 0x7f16d8003520 con 0x7f16f015d330 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 >> v2:172.21.15.121:3300/0 conn(0x7f16f00d2910 msgr2=0x7f16f00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 --2- 172.21.15.46:0/62630796 >> v2:172.21.15.121:3300/0 conn(0x7f16f00d2910 0x7f16f00d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 >> v1:172.21.15.121:6789/0 conn(0x7f16f00d3730 legacy=0x7f16f00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00d1870 con 0x7f16f015d330 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f16e0001670 con 0x7f16f015d330 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 >> v1:172.21.15.46:6789/0 conn(0x7f16f015d330 legacy=0x7f16f00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d8003970 0x7f16d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.881 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d80047c0 0x7f16d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00d3bc0 con 0x7f16d8005180 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d800c120 con 0x7f16d80047c0 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16d800c2e0 con 0x7f16d8003970 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16f4dec640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16effff640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d8003970 0x7f16d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.878+0000 7f16ef7fe640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d80047c0 0x7f16d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 -- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d8003970 msgr2=0x7f16d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d8003970 0x7f16d8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 -- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d80047c0 msgr2=0x7f16d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d80047c0 0x7f16d8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 -- 172.21.15.46:0/62630796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d8003520 con 0x7f16d8005180 2023-12-16T21:15:45.882 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 -- 172.21.15.46:0/62630796 reap_dead start 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16f4dec640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16d800b980 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f16e8001e50 tx=0x7f16d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b92e0132137391 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8003b70 con 0x7f16d8005180 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e8003d10 con 0x7f16d8005180 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.879+0000 7f16eeffd640 1 -- 172.21.15.46:0/62630796 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8005310 con 0x7f16d8005180 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 -- 172.21.15.46:0/62630796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 msgr2=0x7f16d800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16d800b980 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f16e8001e50 tx=0x7f16d80057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 -- 172.21.15.46:0/62630796 shutdown_connections 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 --2- 172.21.15.46:0/62630796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16d800b980 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 -- 172.21.15.46:0/62630796 >> 172.21.15.46:0/62630796 conn(0x7f16f0072020 msgr2=0x7f16f0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 -- 172.21.15.46:0/62630796 shutdown_connections 2023-12-16T21:15:45.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.880+0000 7f16f684f640 1 -- 172.21.15.46:0/62630796 wait complete. 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.881+0000 7f16f684f640 1 Processor -- start 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.881+0000 7f16f684f640 1 -- start start 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16f684f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16f00cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16f684f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f00c7800 0x7f16f01526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16f684f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16f00ca330 0x7f16f00ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16f684f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00cacf0 con 0x7f16d8005180 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16f684f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00c8b60 con 0x7f16f00c7800 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16f684f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f00cb080 con 0x7f16f00ca330 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16f00cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16effff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16f00cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37354/0 (socket says 172.21.15.46:37354) 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.882+0000 7f16effff640 1 -- 172.21.15.46:0/1432239615 learned_addr learned my addr 172.21.15.46:0/1432239615 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16effff640 1 -- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16f00ca330 msgr2=0x7f16f00ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16effff640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16f00ca330 0x7f16f00ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16effff640 1 -- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f00c7800 msgr2=0x7f16f01526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16effff640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f00c7800 0x7f16f01526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16effff640 1 -- 172.21.15.46:0/1432239615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f00c7c80 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16effff640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16f00cc4e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f16e0001e60 tx=0x7f16e0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e000f9c0 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e000fb40 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f00c7f50 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e00133f0 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.883+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f00c8510 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.884+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f0069400 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.884+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f16e0013550 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.885+0000 7f16ecff9640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f16dc06ae70 0x7f16dc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.885+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16e00896a0 con 0x7f16d8005180 2023-12-16T21:15:45.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.885+0000 7f16ef7fe640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f16dc06ae70 0x7f16dc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.886+0000 7f16ef7fe640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f16dc06ae70 0x7f16dc06d330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f16f00b4b00 tx=0x7f16e4009320 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:45.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:45.889+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f16e00547e0 con 0x7f16d8005180 2023-12-16T21:15:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.064+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16f015cd80 con 0x7f16d8005180 2023-12-16T21:15:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.064+0000 7f16ecff9640 1 -- 172.21.15.46:0/1432239615 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f16e0057e00 con 0x7f16d8005180 2023-12-16T21:15:46.064 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f16dc06ae70 msgr2=0x7f16dc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f16dc06ae70 0x7f16dc06d330 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f16f00b4b00 tx=0x7f16e4009320 comp rx=0 tx=0).stop 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 msgr2=0x7f16f00cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16f00cc4e0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f16e0001e60 tx=0x7f16e0007720 comp rx=0 tx=0).stop 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 shutdown_connections 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f16dc06ae70 0x7f16dc06d330 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16f00ca330 0x7f16f00ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16f00c7800 0x7f16f01526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 --2- 172.21.15.46:0/1432239615 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16d8005180 0x7f16f00cc4e0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 >> 172.21.15.46:0/1432239615 conn(0x7f16f0072020 msgr2=0x7f16f00b6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 shutdown_connections 2023-12-16T21:15:46.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.066+0000 7f16f684f640 1 -- 172.21.15.46:0/1432239615 wait complete. 2023-12-16T21:15:46.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:46.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.185+0000 7fc1cbbf5640 1 Processor -- start 2023-12-16T21:15:46.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.186+0000 7fc1cbbf5640 1 -- start start 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.186+0000 7fc1cbbf5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1c415d330 0x7fc1c40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.186+0000 7fc1cbbf5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc1c40d3900 0x7fc1c40b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.186+0000 7fc1cbbf5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c40b2b10 con 0x7fc1c40d2910 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.186+0000 7fc1cbbf5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c40d1590 con 0x7fc1c415d330 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.186+0000 7fc1cbbf5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c40d1870 con 0x7fc1c40d3900 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9991640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc1c415d330 0x7fc1c40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9190640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fc1c40d2910 0x7fc1c40d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56488/0 (socket says 172.21.15.46:56488) 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9190640 1 -- 172.21.15.46:0/2445811665 learned_addr learned my addr 172.21.15.46:0/2445811665 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9991640 1 -- 172.21.15.46:0/2445811665 >> v2:172.21.15.121:3300/0 conn(0x7fc1c40d3900 msgr2=0x7fc1c40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9991640 1 --2- 172.21.15.46:0/2445811665 >> v2:172.21.15.121:3300/0 conn(0x7fc1c40d3900 0x7fc1c40b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9991640 1 -- 172.21.15.46:0/2445811665 >> v1:172.21.15.46:6789/0 conn(0x7fc1c40d2910 legacy=0x7fc1c40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c9991640 1 -- 172.21.15.46:0/2445811665 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40d1b50 con 0x7fc1c415d330 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.187+0000 7fc1c898f640 1 -- 172.21.15.46:0/2445811665 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36850790 0 0) 0x7fc1c40b2b10 con 0x7fc1c40d2910 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1c9991640 1 --2- 172.21.15.46:0/2445811665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c415d330 0x7fc1c40b8b80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc1c406c500 tx=0x7fc1bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fdfb8c7f276886 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1c898f640 1 -- 172.21.15.46:0/2445811665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc02e070 con 0x7fc1c415d330 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1c898f640 1 -- 172.21.15.46:0/2445811665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1bc0315a0 con 0x7fc1c415d330 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1c898f640 1 -- 172.21.15.46:0/2445811665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc031930 con 0x7fc1c415d330 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/2445811665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c415d330 msgr2=0x7fc1c40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/2445811665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c415d330 0x7fc1c40b8b80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fc1c406c500 tx=0x7fc1bc02b840 comp rx=0 tx=0).stop 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/2445811665 shutdown_connections 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/2445811665 >> v2:172.21.15.121:3300/0 conn(0x7fc1c40d3900 0x7fc1c40b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/2445811665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c415d330 0x7fc1c40b8b80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/2445811665 >> 172.21.15.46:0/2445811665 conn(0x7fc1c4072020 msgr2=0x7fc1c4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.188+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/2445811665 shutdown_connections 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.189+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/2445811665 wait complete. 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 Processor -- start 2023-12-16T21:15:46.191 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 -- start start 2023-12-16T21:15:46.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1c40d2910 0x7fc1c40b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 0x7fc1c414eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1c415d330 0x7fc1c414daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c414c1f0 con 0x7fc1c40d3900 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c414ca50 con 0x7fc1c415d330 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1cbbf5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c414cd30 con 0x7fc1c40d2910 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.190+0000 7fc1c9190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 0x7fc1c414eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 0x7fc1c414eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37380/0 (socket says 172.21.15.46:37380) 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 -- 172.21.15.46:0/3970776998 learned_addr learned my addr 172.21.15.46:0/3970776998 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9991640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1c40d2910 0x7fc1c40b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1ca192640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1c415d330 0x7fc1c414daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 -- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1c40d2910 msgr2=0x7fc1c40b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1c40d2910 0x7fc1c40b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 -- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1c415d330 msgr2=0x7fc1c414daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1c415d330 0x7fc1c414daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 -- 172.21.15.46:0/3970776998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1bc009400 con 0x7fc1c40d3900 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c9190640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 0x7fc1c414eef0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c430 tx=0x7fc1b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80074e0 con 0x7fc1c40d3900 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b8007660 con 0x7fc1c40d3900 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c414d010 con 0x7fc1c40d3900 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8004a20 con 0x7fc1c40d3900 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.191+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c414d5d0 con 0x7fc1c40d3900 2023-12-16T21:15:46.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.192+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c414d7f0 con 0x7fc1c40d3900 2023-12-16T21:15:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.193+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc1b8004b80 con 0x7fc1c40d3900 2023-12-16T21:15:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.193+0000 7fc1c27fc640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1b006ae40 0x7fc1b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.193+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1b80892e0 con 0x7fc1c40d3900 2023-12-16T21:15:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.193+0000 7fc1c9991640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1b006ae40 0x7fc1b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.194+0000 7fc1c9991640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1b006ae40 0x7fc1b006d300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc036000 tx=0x7fc1bc03e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.197+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc1b80544d0 con 0x7fc1c40d3900 2023-12-16T21:15:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.387+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc1c40b75d0 con 0x7fc1c40d3900 2023-12-16T21:15:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.388+0000 7fc1c27fc640 1 -- 172.21.15.46:0/3970776998 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc1b8057af0 con 0x7fc1c40d3900 2023-12-16T21:15:46.387 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:46.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1b006ae40 msgr2=0x7fc1b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.390 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1b006ae40 0x7fc1b006d300 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc036000 tx=0x7fc1bc03e040 comp rx=0 tx=0).stop 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 msgr2=0x7fc1c414eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 0x7fc1c414eef0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800c430 tx=0x7fc1b800c8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 shutdown_connections 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc1b006ae40 0x7fc1b006d300 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc1c40d3900 0x7fc1c414eef0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1c415d330 0x7fc1c414daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 --2- 172.21.15.46:0/3970776998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1c40d2910 0x7fc1c40b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 >> 172.21.15.46:0/3970776998 conn(0x7fc1c4072020 msgr2=0x7fc1c40b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:46.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 shutdown_connections 2023-12-16T21:15:46.392 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.390+0000 7fc1cbbf5640 1 -- 172.21.15.46:0/3970776998 wait complete. 2023-12-16T21:15:46.400 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:46.401 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:46.417 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:46.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.564+0000 7f7ece73e640 1 Processor -- start 2023-12-16T21:15:46.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.564+0000 7f7ece73e640 1 -- start start 2023-12-16T21:15:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ece73e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec80b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ece73e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7ec8160f70 0x7f7ec80cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ece73e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ec80ce480 0x7f7ec806b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ece73e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec81614e0 con 0x7f7ec80ce480 2023-12-16T21:15:46.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ece73e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec81617c0 con 0x7f7ec8160f70 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ece73e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8161aa0 con 0x7f7ec815d2d0 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ecccdb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ec80ce480 0x7f7ec806b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ecccdb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7ec80ce480 0x7f7ec806b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37390/0 (socket says 172.21.15.46:37390) 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.565+0000 7f7ecccdb640 1 -- 172.21.15.46:0/3899080600 learned_addr learned my addr 172.21.15.46:0/3899080600 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/3899080600 >> v2:172.21.15.121:3300/0 conn(0x7f7ec8160f70 0x7f7ec80cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/3899080600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec80b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ecccdb640 1 -- 172.21.15.46:0/3899080600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 msgr2=0x7f7ec80b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ecccdb640 1 --2- 172.21.15.46:0/3899080600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec80b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ecccdb640 1 -- 172.21.15.46:0/3899080600 >> v2:172.21.15.121:3300/0 conn(0x7f7ec8160f70 msgr2=0x7f7ec80cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ecccdb640 1 --2- 172.21.15.46:0/3899080600 >> v2:172.21.15.121:3300/0 conn(0x7f7ec8160f70 0x7f7ec80cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.566+0000 7f7ecccdb640 1 -- 172.21.15.46:0/3899080600 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8161d80 con 0x7f7ec80ce480 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ecccdb640 1 --2- 172.21.15.46:0/3899080600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec80ce480 0x7f7ec806b570 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80d4260 tx=0x7f7ec002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac546fdb05adc89 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.567 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ec6ffd640 1 -- 172.21.15.46:0/3899080600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec002ca70 con 0x7f7ec80ce480 2023-12-16T21:15:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ec6ffd640 1 -- 172.21.15.46:0/3899080600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec002cbd0 con 0x7f7ec80ce480 2023-12-16T21:15:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ec6ffd640 1 -- 172.21.15.46:0/3899080600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec00343f0 con 0x7f7ec80ce480 2023-12-16T21:15:46.568 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 -- 172.21.15.46:0/3899080600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec80ce480 msgr2=0x7f7ec806b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 --2- 172.21.15.46:0/3899080600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec80ce480 0x7f7ec806b570 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80d4260 tx=0x7f7ec002bed0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 -- 172.21.15.46:0/3899080600 shutdown_connections 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 --2- 172.21.15.46:0/3899080600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec80ce480 0x7f7ec806b570 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 --2- 172.21.15.46:0/3899080600 >> v2:172.21.15.121:3300/0 conn(0x7f7ec8160f70 0x7f7ec80cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 --2- 172.21.15.46:0/3899080600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec80b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 -- 172.21.15.46:0/3899080600 >> 172.21.15.46:0/3899080600 conn(0x7f7ec8072230 msgr2=0x7f7ec80cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 -- 172.21.15.46:0/3899080600 shutdown_connections 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.567+0000 7f7ece73e640 1 -- 172.21.15.46:0/3899080600 wait complete. 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 Processor -- start 2023-12-16T21:15:46.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 -- start start 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ec80ce480 0x7f7ec80c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec81604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 0x7f7ec81f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec8160a20 con 0x7f7ec8160f70 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec815dbb0 con 0x7f7ec80ce480 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.569+0000 7f7ece73e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec80c9210 con 0x7f7ec815d2d0 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 0x7f7ec81f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 0x7f7ec81f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37396/0 (socket says 172.21.15.46:37396) 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 -- 172.21.15.46:0/186908352 learned_addr learned my addr 172.21.15.46:0/186908352 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ec77fe640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec81604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ec80ce480 0x7f7ec80c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 -- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 msgr2=0x7f7ec81604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec81604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 -- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ec80ce480 msgr2=0x7f7ec80c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ec80ce480 0x7f7ec80c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 -- 172.21.15.46:0/186908352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec0009b30 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ec80ce480 0x7f7ec80c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ecccdb640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 0x7f7ec81f2570 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0036040 tx=0x7f7ec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0009510 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec003e450 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec80c94f0 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec003e800 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.570+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec80c9ab0 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.571+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec815dd50 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.572+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7ec003e960 con 0x7f7ec8160f70 2023-12-16T21:15:46.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.572+0000 7f7ec4ff9640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7eb406ae70 0x7f7eb406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.572+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ec00abec0 con 0x7f7ec8160f70 2023-12-16T21:15:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.573+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7eb406ae70 0x7f7eb406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.573+0000 7f7ec7fff640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7eb406ae70 0x7f7eb406d330 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009b10 tx=0x7f7ebc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.577+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7ec0077000 con 0x7f7ec8160f70 2023-12-16T21:15:46.758 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.759+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec8069400 con 0x7f7ec8160f70 2023-12-16T21:15:46.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.760+0000 7f7ec4ff9640 1 -- 172.21.15.46:0/186908352 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f7ec007a620 con 0x7f7ec8160f70 2023-12-16T21:15:46.760 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:46.760 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7eb406ae70 msgr2=0x7f7eb406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7eb406ae70 0x7f7eb406d330 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009b10 tx=0x7f7ebc008040 comp rx=0 tx=0).stop 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 msgr2=0x7f7ec81f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 0x7f7ec81f2570 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0036040 tx=0x7f7ec0004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 shutdown_connections 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7eb406ae70 0x7f7eb406d330 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7ec8160f70 0x7f7ec81f2570 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ec815d2d0 0x7f7ec81604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.762 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 --2- 172.21.15.46:0/186908352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ec80ce480 0x7f7ec80c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 >> 172.21.15.46:0/186908352 conn(0x7f7ec8072230 msgr2=0x7f7ec80d3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:46.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 shutdown_connections 2023-12-16T21:15:46.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.762+0000 7f7ece73e640 1 -- 172.21.15.46:0/186908352 wait complete. 2023-12-16T21:15:46.773 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:15:29.641456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:15:30.976721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:46.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:46.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.883+0000 7f71e88ee640 1 Processor -- start 2023-12-16T21:15:46.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.883+0000 7f71e88ee640 1 -- start start 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e88ee640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71e406ea50 0x7f71e406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e88ee640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4161d20 con 0x7f71e40d0ed0 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e88ee640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e406f370 con 0x7f71e4160fb0 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e88ee640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e415e760 con 0x7f71e406ea50 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f71e406ea50 0x7f71e406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f71e40d0ed0 0x7f71e40d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:56492/0 (socket says 172.21.15.46:56492) 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e2d9d640 1 -- 172.21.15.46:0/3730270601 learned_addr learned my addr 172.21.15.46:0/3730270601 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e259c640 1 -- 172.21.15.46:0/3730270601 >> v1:172.21.15.121:6789/0 conn(0x7f71e4160fb0 legacy=0x7f71e4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e259c640 1 -- 172.21.15.46:0/3730270601 >> v1:172.21.15.46:6789/0 conn(0x7f71e40d0ed0 legacy=0x7f71e40d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e1d9b640 1 -- 172.21.15.46:0/3730270601 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317017271 0 0) 0x7f71e4161d20 con 0x7f71e40d0ed0 2023-12-16T21:15:46.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.884+0000 7f71e259c640 1 -- 172.21.15.46:0/3730270601 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e415d9a0 con 0x7f71e406ea50 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.885+0000 7f71e259c640 1 --2- 172.21.15.46:0/3730270601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e406ea50 0x7f71e406ee30 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f71d40097a0 tx=0x7f71d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a74f1349d64ca0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.885+0000 7f71e1d9b640 1 -- 172.21.15.46:0/3730270601 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d402e070 con 0x7f71e406ea50 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.885+0000 7f71e1d9b640 1 -- 172.21.15.46:0/3730270601 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d40315a0 con 0x7f71e406ea50 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.885+0000 7f71e1d9b640 1 -- 172.21.15.46:0/3730270601 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4031930 con 0x7f71e406ea50 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 -- 172.21.15.46:0/3730270601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e406ea50 msgr2=0x7f71e406ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 --2- 172.21.15.46:0/3730270601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e406ea50 0x7f71e406ee30 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f71d40097a0 tx=0x7f71d402b840 comp rx=0 tx=0).stop 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 -- 172.21.15.46:0/3730270601 shutdown_connections 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 --2- 172.21.15.46:0/3730270601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e406ea50 0x7f71e406ee30 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 -- 172.21.15.46:0/3730270601 >> 172.21.15.46:0/3730270601 conn(0x7f71e406a6b0 msgr2=0x7f71e406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 -- 172.21.15.46:0/3730270601 shutdown_connections 2023-12-16T21:15:46.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.886+0000 7f71e88ee640 1 -- 172.21.15.46:0/3730270601 wait complete. 2023-12-16T21:15:46.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.887+0000 7f71e88ee640 1 Processor -- start 2023-12-16T21:15:46.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.887+0000 7f71e88ee640 1 -- start start 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e88ee640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71e406ea50 0x7f71e41f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e88ee640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71e40d0ed0 0x7f71e41f3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e88ee640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 0x7f71e41f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e88ee640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41e6c70 con 0x7f71e4160fb0 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e88ee640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e415a6c0 con 0x7f71e406ea50 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e88ee640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41e6e80 con 0x7f71e40d0ed0 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 0x7f71e41f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 0x7f71e41f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37416/0 (socket says 172.21.15.46:37416) 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 -- 172.21.15.46:0/1314361017 learned_addr learned my addr 172.21.15.46:0/1314361017 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 -- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71e40d0ed0 msgr2=0x7f71e41f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71e40d0ed0 0x7f71e41f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.889 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 -- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71e406ea50 msgr2=0x7f71e41f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e259c640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71e406ea50 0x7f71e41f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71e406ea50 0x7f71e41f63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.888+0000 7f71e2d9d640 1 -- 172.21.15.46:0/1314361017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d4009400 con 0x7f71e4160fb0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.889+0000 7f71e2d9d640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 0x7f71e41f3bb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c800 tx=0x7f71dc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.889+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc004d60 con 0x7f71e4160fb0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.889+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc00a640 con 0x7f71e4160fb0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.889+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc00aa30 con 0x7f71e4160fb0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.889+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e41e7160 con 0x7f71e4160fb0 2023-12-16T21:15:46.890 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.889+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41f68e0 con 0x7f71e4160fb0 2023-12-16T21:15:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.890+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e41e7680 con 0x7f71e4160fb0 2023-12-16T21:15:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.890+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f71dc00ab90 con 0x7f71e4160fb0 2023-12-16T21:15:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.891+0000 7f71daffd640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71c406ae40 0x7f71c406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.891+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71dc088ac0 con 0x7f71e4160fb0 2023-12-16T21:15:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.891+0000 7f71e259c640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71c406ae40 0x7f71c406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:46.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.892+0000 7f71e259c640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71c406ae40 0x7f71c406d300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f71d4036000 tx=0x7f71d4008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:46.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:46.895+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f71e41e7680 con 0x7f71e4160fb0 2023-12-16T21:15:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.069+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71e41e7680 con 0x7f71e4160fb0 2023-12-16T21:15:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.070+0000 7f71daffd640 1 -- 172.21.15.46:0/1314361017 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71e41e7680 con 0x7f71e4160fb0 2023-12-16T21:15:47.069 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71c406ae40 msgr2=0x7f71c406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71c406ae40 0x7f71c406d300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f71d4036000 tx=0x7f71d4008030 comp rx=0 tx=0).stop 2023-12-16T21:15:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 msgr2=0x7f71e41f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 0x7f71e41f3bb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f71dc00c800 tx=0x7f71dc00ccc0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 shutdown_connections 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f71c406ae40 0x7f71c406d300 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f71e4160fb0 0x7f71e41f3bb0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71e40d0ed0 0x7f71e41f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 --2- 172.21.15.46:0/1314361017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f71e406ea50 0x7f71e41f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.071+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 >> 172.21.15.46:0/1314361017 conn(0x7f71e406a6b0 msgr2=0x7f71e40b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.072+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 shutdown_connections 2023-12-16T21:15:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.072+0000 7f71e88ee640 1 -- 172.21.15.46:0/1314361017 wait complete. 2023-12-16T21:15:47.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.192+0000 7fdd85815640 1 Processor -- start 2023-12-16T21:15:47.192 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.192+0000 7fdd85815640 1 -- start start 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd85815640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd8006ea50 0x7fdd8006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd85815640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd800d0ed0 0x7fdd800d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd85815640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd80161d20 con 0x7fdd80160fb0 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd85815640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8006f370 con 0x7fdd800d0ed0 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd85815640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd8015e760 con 0x7fdd8006ea50 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd800d0ed0 0x7fdd800d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fdd800d0ed0 0x7fdd800d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37418/0 (socket says 172.21.15.46:37418) 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 -- 172.21.15.46:0/3590688508 learned_addr learned my addr 172.21.15.46:0/3590688508 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7effd640 1 --2- 172.21.15.46:0/3590688508 >> v2:172.21.15.121:3300/0 conn(0x7fdd8006ea50 0x7fdd8006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 -- 172.21.15.46:0/3590688508 >> v2:172.21.15.121:3300/0 conn(0x7fdd8006ea50 msgr2=0x7fdd8006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:47.194 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 --2- 172.21.15.46:0/3590688508 >> v2:172.21.15.121:3300/0 conn(0x7fdd8006ea50 0x7fdd8006ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 -- 172.21.15.46:0/3590688508 >> v1:172.21.15.121:6789/0 conn(0x7fdd80160fb0 legacy=0x7fdd80161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7f7fe640 1 -- 172.21.15.46:0/3590688508 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8015d9a0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.193+0000 7fdd7effd640 1 --2- 172.21.15.46:0/3590688508 >> v2:172.21.15.121:3300/0 conn(0x7fdd8006ea50 0x7fdd8006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.194+0000 7fdd7f7fe640 1 --2- 172.21.15.46:0/3590688508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd800d1320 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdd8006f680 tx=0x7fdd7002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea177273aa5bf224 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.194+0000 7fdd7dffb640 1 -- 172.21.15.46:0/3590688508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7002c9f0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.194+0000 7fdd7dffb640 1 -- 172.21.15.46:0/3590688508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7002cb50 con 0x7fdd800d0ed0 2023-12-16T21:15:47.195 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.194+0000 7fdd7dffb640 1 -- 172.21.15.46:0/3590688508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd700343f0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 -- 172.21.15.46:0/3590688508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 msgr2=0x7fdd800d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 --2- 172.21.15.46:0/3590688508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd800d1320 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdd8006f680 tx=0x7fdd7002be50 comp rx=0 tx=0).stop 2023-12-16T21:15:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 -- 172.21.15.46:0/3590688508 shutdown_connections 2023-12-16T21:15:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 --2- 172.21.15.46:0/3590688508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd800d1320 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 --2- 172.21.15.46:0/3590688508 >> v2:172.21.15.121:3300/0 conn(0x7fdd8006ea50 0x7fdd8006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 -- 172.21.15.46:0/3590688508 >> 172.21.15.46:0/3590688508 conn(0x7fdd8006a6b0 msgr2=0x7fdd8006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 -- 172.21.15.46:0/3590688508 shutdown_connections 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.195+0000 7fdd85815640 1 -- 172.21.15.46:0/3590688508 wait complete. 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.196+0000 7fdd85815640 1 Processor -- start 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.196+0000 7fdd85815640 1 -- start start 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd85815640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8006ea50 0x7fdd8006bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd85815640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd80157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd85815640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd80160fb0 0x7fdd8015a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd85815640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd80158960 con 0x7fdd800d0ed0 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd85815640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd80158c40 con 0x7fdd8006ea50 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd85815640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd80158f20 con 0x7fdd80160fb0 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd80157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.200 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd80157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37422/0 (socket says 172.21.15.46:37422) 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 -- 172.21.15.46:0/1407167254 learned_addr learned my addr 172.21.15.46:0/1407167254 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7effd640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8006ea50 0x7fdd8006bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7f7fe640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd80160fb0 0x7fdd8015a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 -- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd80160fb0 msgr2=0x7fdd8015a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd80160fb0 0x7fdd8015a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 -- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8006ea50 msgr2=0x7fdd8006bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8006ea50 0x7fdd8006bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7e7fc640 1 -- 172.21.15.46:0/1407167254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70009b30 con 0x7fdd800d0ed0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.197+0000 7fdd7effd640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8006ea50 0x7fdd8006bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.198+0000 7fdd7e7fc640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd80157ec0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800c430 tx=0x7fdd6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.198+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd68011790 con 0x7fdd800d0ed0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.198+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd680118f0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.198+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd80159200 con 0x7fdd800d0ed0 2023-12-16T21:15:47.201 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.198+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd68011ce0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.198+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8013f390 con 0x7fdd800d0ed0 2023-12-16T21:15:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.199+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd80069400 con 0x7fdd800d0ed0 2023-12-16T21:15:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.199+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fdd6801c460 con 0x7fdd800d0ed0 2023-12-16T21:15:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.200+0000 7fdd77fff640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdd6006ae40 0x7fdd6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.200+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd680895d0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.203 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.200+0000 7fdd7effd640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdd6006ae40 0x7fdd6006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.204 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.200+0000 7fdd7effd640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdd6006ae40 0x7fdd6006d300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdd8015d170 tx=0x7fdd78008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.204+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fdd68054680 con 0x7fdd800d0ed0 2023-12-16T21:15:47.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.380+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdd8015b200 con 0x7fdd800d0ed0 2023-12-16T21:15:47.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.380+0000 7fdd77fff640 1 -- 172.21.15.46:0/1407167254 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdd68057ca0 con 0x7fdd800d0ed0 2023-12-16T21:15:47.408 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdd6006ae40 msgr2=0x7fdd6006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdd6006ae40 0x7fdd6006d300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fdd8015d170 tx=0x7fdd78008040 comp rx=0 tx=0).stop 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 msgr2=0x7fdd80157ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd80157ec0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800c430 tx=0x7fdd6800c8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 shutdown_connections 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fdd6006ae40 0x7fdd6006d300 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd800d0ed0 0x7fdd80157ec0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd80160fb0 0x7fdd8015a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 --2- 172.21.15.46:0/1407167254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd8006ea50 0x7fdd8006bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 >> 172.21.15.46:0/1407167254 conn(0x7fdd8006a6b0 msgr2=0x7fdd800b1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 shutdown_connections 2023-12-16T21:15:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.382+0000 7fdd85815640 1 -- 172.21.15.46:0/1407167254 wait complete. 2023-12-16T21:15:47.410 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.520+0000 7f1d2a851640 1 Processor -- start 2023-12-16T21:15:47.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.520+0000 7f1d2a851640 1 -- start start 2023-12-16T21:15:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d2a851640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d2415d1d0 0x7f1d2406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d2a851640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d240ad640 0x7f1d2406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d2a851640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d240aeb80 0x7f1d24160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d2a851640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d240adae0 con 0x7f1d240ad640 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d2a851640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d240addc0 con 0x7f1d2415d1d0 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d2a851640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d240ae0a0 con 0x7f1d240aeb80 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d237fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d240ad640 0x7f1d2406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d237fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f1d240ad640 0x7f1d2406ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37434/0 (socket says 172.21.15.46:37434) 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d237fe640 1 -- 172.21.15.46:0/2340079589 learned_addr learned my addr 172.21.15.46:0/2340079589 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d28dee640 1 --2- 172.21.15.46:0/2340079589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d240aeb80 0x7f1d24160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d237fe640 1 -- 172.21.15.46:0/2340079589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d240aeb80 msgr2=0x7f1d24160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d237fe640 1 --2- 172.21.15.46:0/2340079589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d240aeb80 0x7f1d24160ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d23fff640 1 --2- 172.21.15.46:0/2340079589 >> v2:172.21.15.121:3300/0 conn(0x7f1d2415d1d0 0x7f1d2406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.521+0000 7f1d237fe640 1 -- 172.21.15.46:0/2340079589 >> v2:172.21.15.121:3300/0 conn(0x7f1d2415d1d0 msgr2=0x7f1d2406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d237fe640 1 --2- 172.21.15.46:0/2340079589 >> v2:172.21.15.121:3300/0 conn(0x7f1d2415d1d0 0x7f1d2406b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d237fe640 1 -- 172.21.15.46:0/2340079589 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d240ae380 con 0x7f1d240ad640 2023-12-16T21:15:47.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d23fff640 1 --2- 172.21.15.46:0/2340079589 >> v2:172.21.15.121:3300/0 conn(0x7f1d2415d1d0 0x7f1d2406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d237fe640 1 --2- 172.21.15.46:0/2340079589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d2406ba20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1d24072310 tx=0x7f1d1402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fad1d357dfe9d2c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d22ffd640 1 -- 172.21.15.46:0/2340079589 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14031860 con 0x7f1d240ad640 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d22ffd640 1 -- 172.21.15.46:0/2340079589 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d140319c0 con 0x7f1d240ad640 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.522+0000 7f1d22ffd640 1 -- 172.21.15.46:0/2340079589 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d14031d50 con 0x7f1d240ad640 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 -- 172.21.15.46:0/2340079589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 msgr2=0x7f1d2406ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2340079589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d2406ba20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f1d24072310 tx=0x7f1d1402b900 comp rx=0 tx=0).stop 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 -- 172.21.15.46:0/2340079589 shutdown_connections 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2340079589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d240aeb80 0x7f1d24160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2340079589 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d2406ba20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2340079589 >> v2:172.21.15.121:3300/0 conn(0x7f1d2415d1d0 0x7f1d2406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 -- 172.21.15.46:0/2340079589 >> 172.21.15.46:0/2340079589 conn(0x7f1d24070e70 msgr2=0x7f1d24071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 -- 172.21.15.46:0/2340079589 shutdown_connections 2023-12-16T21:15:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.523+0000 7f1d2a851640 1 -- 172.21.15.46:0/2340079589 wait complete. 2023-12-16T21:15:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.524+0000 7f1d2a851640 1 Processor -- start 2023-12-16T21:15:47.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.524+0000 7f1d2a851640 1 -- start start 2023-12-16T21:15:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d2a851640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d240ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d2a851640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d240aeb80 0x7f1d240cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d2a851640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2415d1d0 0x7f1d241f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d2a851640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d24149720 con 0x7f1d240ad640 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d2a851640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d241607d0 con 0x7f1d240aeb80 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d2a851640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d24149970 con 0x7f1d2415d1d0 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d240ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d240ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37450/0 (socket says 172.21.15.46:37450) 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 -- 172.21.15.46:0/2737470420 learned_addr learned my addr 172.21.15.46:0/2737470420 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d237fe640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d240aeb80 0x7f1d240cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d28dee640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2415d1d0 0x7f1d241f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 -- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2415d1d0 msgr2=0x7f1d241f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2415d1d0 0x7f1d241f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 -- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d240aeb80 msgr2=0x7f1d240cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d240aeb80 0x7f1d240cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.525+0000 7f1d23fff640 1 -- 172.21.15.46:0/2737470420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d140094d0 con 0x7f1d240ad640 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.526+0000 7f1d23fff640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d240ce550 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c820 tx=0x7f1d0c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.526+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c007a30 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.526+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c007b90 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.526+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d24160180 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.526+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00ce10 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.526+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2415fe70 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.527+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d24069400 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.527+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1d0c0113f0 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.528+0000 7f1d20ff9640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d0406ae40 0x7f1d0406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.528+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d0c089740 con 0x7f1d240ad640 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.528+0000 7f1d237fe640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d0406ae40 0x7f1d0406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.528+0000 7f1d237fe640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d0406ae40 0x7f1d0406d300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1d14036000 tx=0x7f1d1403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.532+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f1d0c0547f0 con 0x7f1d240ad640 2023-12-16T21:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.710+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1d2415a2a0 con 0x7f1d240ad640 2023-12-16T21:15:47.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.711+0000 7f1d20ff9640 1 -- 172.21.15.46:0/2737470420 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1d0c057e10 con 0x7f1d240ad640 2023-12-16T21:15:47.711 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d0406ae40 msgr2=0x7f1d0406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d0406ae40 0x7f1d0406d300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1d14036000 tx=0x7f1d1403c040 comp rx=0 tx=0).stop 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 msgr2=0x7f1d240ce550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d240ce550 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00c820 tx=0x7f1d0c00cce0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 shutdown_connections 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f1d0406ae40 0x7f1d0406d300 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d2415d1d0 0x7f1d241f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d240aeb80 0x7f1d240cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 --2- 172.21.15.46:0/2737470420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d240ad640 0x7f1d240ce550 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 >> 172.21.15.46:0/2737470420 conn(0x7f1d24070e70 msgr2=0x7f1d2406a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 shutdown_connections 2023-12-16T21:15:47.714 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.713+0000 7f1d2a851640 1 -- 172.21.15.46:0/2737470420 wait complete. 2023-12-16T21:15:47.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.832+0000 7f83dcf7c640 1 Processor -- start 2023-12-16T21:15:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.832+0000 7f83dcf7c640 1 -- start start 2023-12-16T21:15:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83dcf7c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83d8140310 0x7f83d81618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83dcf7c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81409d0 0x7f83d806a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83dcf7c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d81406f0 con 0x7f83d81428d0 2023-12-16T21:15:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83dcf7c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d8161e20 con 0x7f83d8140310 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83dcf7c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d8143960 con 0x7f83d81409d0 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83d659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83d8140310 0x7f83d81618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83d659c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f83d8140310 0x7f83d81618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37452/0 (socket says 172.21.15.46:37452) 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83d659c640 1 -- 172.21.15.46:0/2914484854 learned_addr learned my addr 172.21.15.46:0/2914484854 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.833+0000 7f83d5d9b640 1 --2- 172.21.15.46:0/2914484854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81409d0 0x7f83d806a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.834+0000 7f83d659c640 1 -- 172.21.15.46:0/2914484854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81409d0 msgr2=0x7f83d806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.834+0000 7f83d659c640 1 --2- 172.21.15.46:0/2914484854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81409d0 0x7f83d806a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.834+0000 7f83d659c640 1 -- 172.21.15.46:0/2914484854 >> v1:172.21.15.121:6789/0 conn(0x7f83d81428d0 legacy=0x7f83d80714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.834+0000 7f83d659c640 1 -- 172.21.15.46:0/2914484854 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8142de0 con 0x7f83d8140310 2023-12-16T21:15:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.834+0000 7f83d659c640 1 --2- 172.21.15.46:0/2914484854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d81618e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009870 tx=0x7f83d002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d538a1f39d46b57 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83d559a640 1 -- 172.21.15.46:0/2914484854 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0031640 con 0x7f83d8140310 2023-12-16T21:15:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83d559a640 1 -- 172.21.15.46:0/2914484854 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d00317a0 con 0x7f83d8140310 2023-12-16T21:15:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83d559a640 1 -- 172.21.15.46:0/2914484854 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0031b30 con 0x7f83d8140310 2023-12-16T21:15:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 -- 172.21.15.46:0/2914484854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 msgr2=0x7f83d81618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/2914484854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d81618e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009870 tx=0x7f83d002b8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 -- 172.21.15.46:0/2914484854 shutdown_connections 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/2914484854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81409d0 0x7f83d806a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/2914484854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d81618e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 -- 172.21.15.46:0/2914484854 >> 172.21.15.46:0/2914484854 conn(0x7f83d80b1e50 msgr2=0x7f83d80b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 -- 172.21.15.46:0/2914484854 shutdown_connections 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.835+0000 7f83dcf7c640 1 -- 172.21.15.46:0/2914484854 wait complete. 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 Processor -- start 2023-12-16T21:15:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 -- start start 2023-12-16T21:15:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d813f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d81409d0 0x7f83d81f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81428d0 0x7f83d80ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d81557f0 con 0x7f83d8140310 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d81f3f50 con 0x7f83d81409d0 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83dcf7c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d81f3990 con 0x7f83d81428d0 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83d659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d813f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83d659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d813f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37468/0 (socket says 172.21.15.46:37468) 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.837+0000 7f83d659c640 1 -- 172.21.15.46:0/1474135923 learned_addr learned my addr 172.21.15.46:0/1474135923 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d5d9b640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d81409d0 0x7f83d81f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d6d9d640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81428d0 0x7f83d80ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d659c640 1 -- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81428d0 msgr2=0x7f83d80ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d659c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81428d0 0x7f83d80ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d659c640 1 -- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d81409d0 msgr2=0x7f83d81f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d659c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d81409d0 0x7f83d81f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d659c640 1 -- 172.21.15.46:0/1474135923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d00094d0 con 0x7f83d8140310 2023-12-16T21:15:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d5d9b640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d81409d0 0x7f83d81f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d6d9d640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81428d0 0x7f83d80ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83d659c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d813f850 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f83d0000c00 tx=0x7f83d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d003a930 con 0x7f83d8140310 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d003aa90 con 0x7f83d8140310 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d003ae40 con 0x7f83d8140310 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d81f3c70 con 0x7f83d8140310 2023-12-16T21:15:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.838+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d81f4510 con 0x7f83d8140310 2023-12-16T21:15:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.839+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d8069400 con 0x7f83d8140310 2023-12-16T21:15:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.840+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f83d002f050 con 0x7f83d8140310 2023-12-16T21:15:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.840+0000 7f83ceffd640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f83b806ae40 0x7f83b806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.840+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83d00ab450 con 0x7f83d8140310 2023-12-16T21:15:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.840+0000 7f83d5d9b640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f83b806ae40 0x7f83b806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.841+0000 7f83d5d9b640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f83b806ae40 0x7f83b806d300 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f83d81f6520 tx=0x7f83c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:47.845+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f83d0076460 con 0x7f83d8140310 2023-12-16T21:15:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.034+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f83d806ea60 con 0x7f83d8140310 2023-12-16T21:15:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.035+0000 7f83ceffd640 1 -- 172.21.15.46:0/1474135923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7f83d0079a80 con 0x7f83d8140310 2023-12-16T21:15:48.034 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.036+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f83b806ae40 msgr2=0x7f83b806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:48.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.036+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f83b806ae40 0x7f83b806d300 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f83d81f6520 tx=0x7f83c0008040 comp rx=0 tx=0).stop 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.036+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 msgr2=0x7f83d813f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.036+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d813f850 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f83d0000c00 tx=0x7f83d00046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 shutdown_connections 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f83b806ae40 0x7f83b806d300 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f83d81428d0 0x7f83d80ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83d81409d0 0x7f83d81f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 --2- 172.21.15.46:0/1474135923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f83d8140310 0x7f83d813f850 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 >> 172.21.15.46:0/1474135923 conn(0x7f83d80b1e50 msgr2=0x7f83d806f640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 shutdown_connections 2023-12-16T21:15:48.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:48.037+0000 7f83dcf7c640 1 -- 172.21.15.46:0/1474135923 wait complete. 2023-12-16T21:15:48.047 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:15:48.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:48.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:48.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[110525]: fuse finished with error 0 and tester_r 0 2023-12-16T21:15:48.078 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:15:54.179 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:54.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.180 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:15:54.200 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-16T21:15:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.360+0000 7fd2d853e640 1 Processor -- start 2023-12-16T21:15:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.360+0000 7fd2d853e640 1 -- start start 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d853e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2d01409d0 0x7fd2d01618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d853e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d01406f0 con 0x7fd2d0140310 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d853e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d0143960 con 0x7fd2d006abe0 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d853e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d0161e20 con 0x7fd2d01409d0 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d6adb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2d01409d0 0x7fd2d01618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d6adb640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd2d01409d0 0x7fd2d01618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48004/0 (socket says 172.21.15.46:48004) 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d6adb640 1 -- 172.21.15.46:0/1665583064 learned_addr learned my addr 172.21.15.46:0/1665583064 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d6adb640 1 -- 172.21.15.46:0/1665583064 >> v1:172.21.15.121:6789/0 conn(0x7fd2d006abe0 legacy=0x7fd2d006efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d6adb640 1 -- 172.21.15.46:0/1665583064 >> v1:172.21.15.46:6789/0 conn(0x7fd2d0140310 legacy=0x7fd2d006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.361+0000 7fd2d6adb640 1 -- 172.21.15.46:0/1665583064 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0142de0 con 0x7fd2d01409d0 2023-12-16T21:15:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.362+0000 7fd2d6adb640 1 --2- 172.21.15.46:0/1665583064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d01618e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009870 tx=0x7fd2c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725a2ef6341b32a6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.362+0000 7fd2d52d8640 1 -- 172.21.15.46:0/1665583064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8030860 con 0x7fd2d01409d0 2023-12-16T21:15:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.362+0000 7fd2d52d8640 1 -- 172.21.15.46:0/1665583064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c80309c0 con 0x7fd2d01409d0 2023-12-16T21:15:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.362+0000 7fd2d52d8640 1 -- 172.21.15.46:0/1665583064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8030d50 con 0x7fd2d01409d0 2023-12-16T21:15:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 -- 172.21.15.46:0/1665583064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 msgr2=0x7fd2d01618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 --2- 172.21.15.46:0/1665583064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d01618e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009870 tx=0x7fd2c802b9a0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 -- 172.21.15.46:0/1665583064 shutdown_connections 2023-12-16T21:15:54.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 --2- 172.21.15.46:0/1665583064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d01618e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 -- 172.21.15.46:0/1665583064 >> 172.21.15.46:0/1665583064 conn(0x7fd2d00b1e50 msgr2=0x7fd2d00b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:54.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 -- 172.21.15.46:0/1665583064 shutdown_connections 2023-12-16T21:15:54.364 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.363+0000 7fd2d853e640 1 -- 172.21.15.46:0/1665583064 wait complete. 2023-12-16T21:15:54.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.364+0000 7fd2d853e640 1 Processor -- start 2023-12-16T21:15:54.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.364+0000 7fd2d853e640 1 -- start start 2023-12-16T21:15:54.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d853e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d006abe0 0x7fd2d00af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:54.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d853e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d0140310 0x7fd2d0145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d853e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d0146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d853e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00aec20 con 0x7fd2d01409d0 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d853e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00ae010 con 0x7fd2d006abe0 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d853e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00ae2f0 con 0x7fd2d0140310 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d0146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d0146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48008/0 (socket says 172.21.15.46:48008) 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 -- 172.21.15.46:0/868983779 learned_addr learned my addr 172.21.15.46:0/868983779 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d62da640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d006abe0 0x7fd2d00af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d5ad9640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d0140310 0x7fd2d0145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:54.367 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 -- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d0140310 msgr2=0x7fd2d0145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d0140310 0x7fd2d0145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 -- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d006abe0 msgr2=0x7fd2d00af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d006abe0 0x7fd2d00af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d6adb640 1 -- 172.21.15.46:0/868983779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c80094d0 con 0x7fd2d01409d0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d5ad9640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d0140310 0x7fd2d0145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.365+0000 7fd2d62da640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d006abe0 0x7fd2d00af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.366+0000 7fd2d6adb640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d0146280 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009870 tx=0x7fd2c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.366+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80307f0 con 0x7fd2d01409d0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.366+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c8043400 con 0x7fd2d01409d0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.366+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d00ae5d0 con 0x7fd2d01409d0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.366+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c80437b0 con 0x7fd2d01409d0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.366+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d0135ba0 con 0x7fd2d01409d0 2023-12-16T21:15:54.368 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.367+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d00aedd0 con 0x7fd2d01409d0 2023-12-16T21:15:54.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.367+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd2c8043910 con 0x7fd2d01409d0 2023-12-16T21:15:54.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.368+0000 7fd2ceffd640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd2bc06ae40 0x7fd2bc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:54.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.368+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2c803e070 con 0x7fd2d01409d0 2023-12-16T21:15:54.369 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.368+0000 7fd2d62da640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd2bc06ae40 0x7fd2bc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:54.370 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.368+0000 7fd2d62da640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd2bc06ae40 0x7fd2bc06d300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00012e0 tx=0x7fd2c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:54.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.372+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd2d00aedd0 con 0x7fd2d01409d0 2023-12-16T21:15:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.588+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd2d0161030 con 0x7fd2d01409d0 2023-12-16T21:15:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.588+0000 7fd2ceffd640 1 -- 172.21.15.46:0/868983779 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd2c8077ab0 con 0x7fd2d01409d0 2023-12-16T21:15:54.588 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-16T21:15:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd2bc06ae40 msgr2=0x7fd2bc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd2bc06ae40 0x7fd2bc06d300 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00012e0 tx=0x7fd2c0009040 comp rx=0 tx=0).stop 2023-12-16T21:15:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 msgr2=0x7fd2d0146280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d0146280 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009870 tx=0x7fd2c8004730 comp rx=0 tx=0).stop 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 shutdown_connections 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd2bc06ae40 0x7fd2bc06d300 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd2d01409d0 0x7fd2d0146280 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d0140310 0x7fd2d0145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 --2- 172.21.15.46:0/868983779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2d006abe0 0x7fd2d00af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 >> 172.21.15.46:0/868983779 conn(0x7fd2d00b1e50 msgr2=0x7fd2d00714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 shutdown_connections 2023-12-16T21:15:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:54.590+0000 7fd2d853e640 1 -- 172.21.15.46:0/868983779 wait complete. 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:15:54.601 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:54.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.602 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2023-12-16T21:15:54.617 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:15:54.617 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:15:54.617 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:15:54.617 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 42530sec preferred_lft 42530sec 2023-12-16T21:15:54.618 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::7c77:b0ff:fec9:41e/64 scope link 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether e6:8b:50:a7:d5:35 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::e48b:50ff:fea7:d535/64 scope link 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.619 INFO:teuthology.orchestra.run.smithi046.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-16T21:15:54.620 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 22:ee:6c:e1:56:15 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:54.620 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::20ee:6cff:fee1:5615/64 scope link 2023-12-16T21:15:54.620 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:15:54.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.620 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:15:54.687 INFO:teuthology.orchestra.run.smithi046.stdout:1 2023-12-16T21:15:54.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.688 DEBUG:teuthology.orchestra.run.smithi046:> route 2023-12-16T21:15:54.745 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2023-12-16T21:15:54.745 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:15:54.745 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:15:54.745 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:15:54.745 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:15:54.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.745 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2023-12-16T21:15:54.746 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2023-12-16T21:15:54.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:15:54.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:15:54.746 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:15:54.746 DEBUG:teuthology.orchestra.run.smithi046:> ') 2023-12-16T21:15:54.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.850 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:15:54.868 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:15:54.868 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:15:54.868 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:54.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:54.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2023-12-16T21:15:54.939 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-16T21:15:54.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:54.939 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:54.995 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:15:54.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:15:54.995 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:55.050 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:55.121 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:55.177 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:55.177 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:55.178 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:55.179 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:55.180 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:55.180 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:55.180 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:55.180 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:55.180 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:55.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:55.180 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:55.232 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-16T21:15:55.233 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-16T21:15:55.276 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2023-12-16T21:15:55.306 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2023-12-16T21:15:55.363 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2023-12-16T21:15:55.363 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2023-12-16T21:15:55.364 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2023-12-16T21:15:55.364 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2023-12-16T21:15:55.364 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2023-12-16T21:15:55.364 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2023-12-16T21:15:55.364 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2023-12-16T21:15:55.364 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2023-12-16T21:15:55.365 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2023-12-16T21:15:55.366 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2023-12-16T21:15:55.366 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2023-12-16T21:15:55.366 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2023-12-16T21:15:55.366 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2023-12-16T21:15:55.366 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2023-12-16T21:15:55.366 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-0 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-1 2023-12-16T21:15:55.367 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-2 2023-12-16T21:15:55.368 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph/osd/ceph-3 2023-12-16T21:15:55.368 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2023-12-16T21:15:55.368 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:55.368 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:15:55.368 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:55.368 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:15:55.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:55.369 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2023-12-16T21:15:55.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:2023-12-16T21:15:55.369+0000 7fe981f67480 -1 init, newargv = 0x55c73e547860 newargc=15 2023-12-16T21:15:55.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[111487]: starting ceph client 2023-12-16T21:15:55.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[111487]: starting fuse 2023-12-16T21:15:55.387 INFO:teuthology.orchestra.run.smithi046.stdout:41 2023-12-16T21:15:55.387 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> import re 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2023-12-16T21:15:55.387 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> return f 2023-12-16T21:15:55.388 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-16T21:15:55.389 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:55.389 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.0")) 2023-12-16T21:15:55.389 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:15:55.512 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.0.111487.asok 2023-12-16T21:15:55.516 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-16T21:15:55.523 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.111487.asok 2023-12-16T21:15:55.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:55.523 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.111487.asok status 2023-12-16T21:15:55.666 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:15:55.666 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "c7586e95519b7be76f3b6e3a4b333888baf46bc9", 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-202-gc7586e95 (c7586e95519b7be76f3b6e3a4b333888baf46bc9) reef (dev)", 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "0", 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "111487", 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 6261, 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2023-12-16T21:15:55.667 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 6261 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1313228396 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: } 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 1313228396 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.6261 192.168.0.1:0/1313228396", 2023-12-16T21:15:55.668 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/1313228396", 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 20, 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 50, 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2023-12-16T21:15:55.669 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:15:55.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:55.675 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2023-12-16T21:15:55.675 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:55.690 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2023-12-16T21:15:55.691 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:55.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:55.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> stdin-killer -- python3 -c ' 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> import os 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> import errno 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> try: 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> if True: 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> ret = os.read(fd, 1024) 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> else: 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> os.write(fd, b'"'"'content'"'"') 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> except OSError as e: 2023-12-16T21:15:55.762 DEBUG:teuthology.orchestra.run.smithi046:> if e.errno != errno.EPERM: 2023-12-16T21:15:55.763 DEBUG:teuthology.orchestra.run.smithi046:> raise 2023-12-16T21:15:55.763 DEBUG:teuthology.orchestra.run.smithi046:> else: 2023-12-16T21:15:55.763 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("client does not check permission of data pool") 2023-12-16T21:15:55.763 DEBUG:teuthology.orchestra.run.smithi046:> ' 2023-12-16T21:15:55.871 INFO:teuthology.orchestra.run.smithi046.stderr:Traceback (most recent call last): 2023-12-16T21:15:55.871 INFO:teuthology.orchestra.run.smithi046.stderr: File "", line 15, in 2023-12-16T21:15:55.871 INFO:teuthology.orchestra.run.smithi046.stderr:RuntimeError: client does not check permission of data pool 2023-12-16T21:15:55.874 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:55 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2023-12-16T21:15:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:55.880 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2023-12-16T21:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.034+0000 7fe65ad05640 1 Processor -- start 2023-12-16T21:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.034+0000 7fe65ad05640 1 -- start start 2023-12-16T21:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.035+0000 7fe65ad05640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe65406bbf0 0x7fe6540b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.035+0000 7fe65ad05640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6541464d0 0x7fe6540ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.035+0000 7fe65ad05640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe654146d50 con 0x7fe654140480 2023-12-16T21:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.035+0000 7fe65ad05640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6540ada30 con 0x7fe65406bbf0 2023-12-16T21:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.035+0000 7fe65ad05640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6540b37c0 con 0x7fe6541464d0 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe65406bbf0 0x7fe6540b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe65406bbf0 0x7fe6540b2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:47656/0 (socket says 172.21.15.46:47656) 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 -- 172.21.15.46:0/213864066 learned_addr learned my addr 172.21.15.46:0/213864066 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe6592a2640 1 --2- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6541464d0 0x7fe6540ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 -- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6541464d0 msgr2=0x7fe6540ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 --2- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6541464d0 0x7fe6540ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 -- 172.21.15.46:0/213864066 >> v1:172.21.15.121:6789/0 conn(0x7fe654140480 legacy=0x7fe65406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.036+0000 7fe653fff640 1 -- 172.21.15.46:0/213864066 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6540ad700 con 0x7fe65406bbf0 2023-12-16T21:15:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.037+0000 7fe653fff640 1 --2- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe65406bbf0 0x7fe6540b2820 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009870 tx=0x7fe63c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab6ac3d4af1515f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.037+0000 7fe6537fe640 1 -- 172.21.15.46:0/213864066 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c031640 con 0x7fe65406bbf0 2023-12-16T21:15:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.037+0000 7fe6537fe640 1 -- 172.21.15.46:0/213864066 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c0317a0 con 0x7fe65406bbf0 2023-12-16T21:15:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.037+0000 7fe6537fe640 1 -- 172.21.15.46:0/213864066 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c031b30 con 0x7fe65406bbf0 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 -- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe65406bbf0 msgr2=0x7fe6540b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 --2- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe65406bbf0 0x7fe6540b2820 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009870 tx=0x7fe63c02b8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 -- 172.21.15.46:0/213864066 shutdown_connections 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 --2- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6541464d0 0x7fe6540ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 --2- 172.21.15.46:0/213864066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe65406bbf0 0x7fe6540b2820 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 -- 172.21.15.46:0/213864066 >> 172.21.15.46:0/213864066 conn(0x7fe654070c70 msgr2=0x7fe654071050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 -- 172.21.15.46:0/213864066 shutdown_connections 2023-12-16T21:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.038+0000 7fe65ad05640 1 -- 172.21.15.46:0/213864066 wait complete. 2023-12-16T21:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.039+0000 7fe65ad05640 1 Processor -- start 2023-12-16T21:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.039+0000 7fe65ad05640 1 -- start start 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe65ad05640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe65406bbf0 0x7fe654145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe65ad05640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 0x7fe6541e7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe65ad05640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6541464d0 0x7fe654160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe65ad05640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6541498e0 con 0x7fe654140480 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe65ad05640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6541495b0 con 0x7fe6541464d0 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe65ad05640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe654145a40 con 0x7fe65406bbf0 2023-12-16T21:15:56.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe653fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 0x7fe6541e7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe653fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 0x7fe6541e7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48018/0 (socket says 172.21.15.46:48018) 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.040+0000 7fe653fff640 1 -- 172.21.15.46:0/1589705977 learned_addr learned my addr 172.21.15.46:0/1589705977 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe6592a2640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6541464d0 0x7fe654160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe658aa1640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe65406bbf0 0x7fe654145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe653fff640 1 -- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe65406bbf0 msgr2=0x7fe654145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe653fff640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe65406bbf0 0x7fe654145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe653fff640 1 -- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6541464d0 msgr2=0x7fe654160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe653fff640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6541464d0 0x7fe654160240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe653fff640 1 -- 172.21.15.46:0/1589705977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0094d0 con 0x7fe654140480 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe6592a2640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6541464d0 0x7fe654160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe658aa1640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe65406bbf0 0x7fe654145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe653fff640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 0x7fe6541e7aa0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe63c000c00 tx=0x7fe63c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c03a930 con 0x7fe654140480 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c03aa90 con 0x7fe654140480 2023-12-16T21:15:56.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6541e84d0 con 0x7fe654140480 2023-12-16T21:15:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c03ae40 con 0x7fe654140480 2023-12-16T21:15:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.041+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6541e8a90 con 0x7fe654140480 2023-12-16T21:15:56.043 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.042+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe654149a90 con 0x7fe654140480 2023-12-16T21:15:56.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.042+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe63c02f050 con 0x7fe654140480 2023-12-16T21:15:56.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.043+0000 7fe6517fa640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe63406ae40 0x7fe63406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.043+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe63c0ab340 con 0x7fe654140480 2023-12-16T21:15:56.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.043+0000 7fe658aa1640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe63406ae40 0x7fe63406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.044 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.044+0000 7fe658aa1640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe63406ae40 0x7fe63406d300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe65415d1d0 tx=0x7fe64c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.047 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.047+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe654149a90 con 0x7fe654140480 2023-12-16T21:15:56.288 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.288+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe654069400 con 0x7fe654140480 2023-12-16T21:15:56.289 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.290+0000 7fe6517fa640 1 -- 172.21.15.46:0/1589705977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v22) v1 ==== 184+0+141 (secure 0 0 0) 0x7fe63c079f10 con 0x7fe654140480 2023-12-16T21:15:56.290 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2023-12-16T21:15:56.290 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQC+En5lkVwtAxAAxfEiydh05ZaKuGZMlXMe0w== 2023-12-16T21:15:56.290 INFO:teuthology.orchestra.run.smithi046.stdout: caps mds = "allow" 2023-12-16T21:15:56.290 INFO:teuthology.orchestra.run.smithi046.stdout: caps mon = "allow r" 2023-12-16T21:15:56.290 INFO:teuthology.orchestra.run.smithi046.stdout: caps osd = "allow rw pool=cephfs_data" 2023-12-16T21:15:56.290 INFO:teuthology.orchestra.run.smithi046.stderr:updated caps for client.0 2023-12-16T21:15:56.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe63406ae40 msgr2=0x7fe63406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe63406ae40 0x7fe63406d300 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe65415d1d0 tx=0x7fe64c009040 comp rx=0 tx=0).stop 2023-12-16T21:15:56.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 msgr2=0x7fe6541e7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 0x7fe6541e7aa0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe63c000c00 tx=0x7fe63c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.292 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 shutdown_connections 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe63406ae40 0x7fe63406d300 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe654140480 0x7fe6541e7aa0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6541464d0 0x7fe654160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 --2- 172.21.15.46:0/1589705977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe65406bbf0 0x7fe654145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 >> 172.21.15.46:0/1589705977 conn(0x7fe654070c70 msgr2=0x7fe6540ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 shutdown_connections 2023-12-16T21:15:56.293 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.292+0000 7fe65ad05640 1 -- 172.21.15.46:0/1589705977 wait complete. 2023-12-16T21:15:56.303 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-16T21:15:56.303 DEBUG:teuthology.orchestra.run.smithi046:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:15:56.346 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-16T21:15:56.398 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-16T21:15:56.398 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:56.398 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:56.414 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.561+0000 7fd7ca3a9640 1 Processor -- start 2023-12-16T21:15:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.561+0000 7fd7ca3a9640 1 -- start start 2023-12-16T21:15:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7ca3a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd7c415d330 0x7fd7c40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7ca3a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7c40d2910 0x7fd7c40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7ca3a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d3730 0x7fd7c40b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7ca3a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c40b2b10 con 0x7fd7c40d2910 2023-12-16T21:15:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7ca3a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c40d3b90 con 0x7fd7c415d330 2023-12-16T21:15:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7ca3a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40d1590 con 0x7fd7c40d3730 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7c40d2910 0x7fd7c40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd7c40d2910 0x7fd7c40d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48028/0 (socket says 172.21.15.46:48028) 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.562+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1456033342 learned_addr learned my addr 172.21.15.46:0/1456033342 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c8946640 1 --2- 172.21.15.46:0/1456033342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d3730 0x7fd7c40b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c3fff640 1 --2- 172.21.15.46:0/1456033342 >> v2:172.21.15.121:3300/0 conn(0x7fd7c415d330 0x7fd7c40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1456033342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d3730 msgr2=0x7fd7c40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/1456033342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d3730 0x7fd7c40b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1456033342 >> v2:172.21.15.121:3300/0 conn(0x7fd7c415d330 msgr2=0x7fd7c40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/1456033342 >> v2:172.21.15.121:3300/0 conn(0x7fd7c415d330 0x7fd7c40b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c37fe640 1 -- 172.21.15.46:0/1456033342 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40d1870 con 0x7fd7c40d2910 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.563+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/1456033342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c40d2910 0x7fd7c40d31f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40d0d10 tx=0x7fd7b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96027b32a1c4d983 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7c2ffd640 1 -- 172.21.15.46:0/1456033342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b80317e0 con 0x7fd7c40d2910 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7c2ffd640 1 -- 172.21.15.46:0/1456033342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b8031940 con 0x7fd7c40d2910 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7c2ffd640 1 -- 172.21.15.46:0/1456033342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8031cd0 con 0x7fd7c40d2910 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1456033342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c40d2910 msgr2=0x7fd7c40d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1456033342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c40d2910 0x7fd7c40d31f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40d0d10 tx=0x7fd7b802b880 comp rx=0 tx=0).stop 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1456033342 shutdown_connections 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1456033342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d3730 0x7fd7c40b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1456033342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c40d2910 0x7fd7c40d31f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1456033342 >> v2:172.21.15.121:3300/0 conn(0x7fd7c415d330 0x7fd7c40b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1456033342 >> 172.21.15.46:0/1456033342 conn(0x7fd7c4072020 msgr2=0x7fd7c4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1456033342 shutdown_connections 2023-12-16T21:15:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.564+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1456033342 wait complete. 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.565+0000 7fd7ca3a9640 1 Processor -- start 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.565+0000 7fd7ca3a9640 1 -- start start 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7ca3a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d2910 0x7fd7c41f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7ca3a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7c40d3730 0x7fd7c41535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7ca3a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 0x7fd7c40cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7ca3a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41f7630 con 0x7fd7c415d330 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7ca3a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40c6ba0 con 0x7fd7c40d3730 2023-12-16T21:15:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7ca3a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c4160510 con 0x7fd7c40d2910 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7c8946640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 0x7fd7c40cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7c8946640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 0x7fd7c40cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48036/0 (socket says 172.21.15.46:48036) 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7c8946640 1 -- 172.21.15.46:0/1568939521 learned_addr learned my addr 172.21.15.46:0/1568939521 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7c3fff640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d2910 0x7fd7c41f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.566+0000 7fd7c37fe640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7c40d3730 0x7fd7c41535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c8946640 1 -- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d2910 msgr2=0x7fd7c41f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c8946640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d2910 0x7fd7c41f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c8946640 1 -- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7c40d3730 msgr2=0x7fd7c41535d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c8946640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7c40d3730 0x7fd7c41535d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c8946640 1 -- 172.21.15.46:0/1568939521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b80094d0 con 0x7fd7c415d330 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c8946640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 0x7fd7c40cc5c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc007e60 tx=0x7fd7bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc014620 con 0x7fd7c415d330 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc00ee90 con 0x7fd7c415d330 2023-12-16T21:15:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c40c6df0 con 0x7fd7c415d330 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0149e0 con 0x7fd7c415d330 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.567+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c40c73b0 con 0x7fd7c415d330 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.568+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c40c75a0 con 0x7fd7c415d330 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.568+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd7bc014b40 con 0x7fd7c415d330 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.569+0000 7fd7c0ff9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7b006ae40 0x7fd7b006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.569+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7bc089d70 con 0x7fd7c415d330 2023-12-16T21:15:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.569+0000 7fd7c3fff640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7b006ae40 0x7fd7b006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.570+0000 7fd7c3fff640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7b006ae40 0x7fd7b006d300 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd7b40079a0 tx=0x7fd7b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.573+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd7bc054f60 con 0x7fd7c415d330 2023-12-16T21:15:56.753 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.753+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7c4069400 con 0x7fd7c415d330 2023-12-16T21:15:56.754 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.754+0000 7fd7c0ff9640 1 -- 172.21.15.46:0/1568939521 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fd7bc058580 con 0x7fd7c415d330 2023-12-16T21:15:56.754 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:56.755 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:56.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7b006ae40 msgr2=0x7fd7b006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7b006ae40 0x7fd7b006d300 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd7b40079a0 tx=0x7fd7b4008040 comp rx=0 tx=0).stop 2023-12-16T21:15:56.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 msgr2=0x7fd7c40cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 0x7fd7c40cc5c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc007e60 tx=0x7fd7bc00e4b0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 shutdown_connections 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd7b006ae40 0x7fd7b006d300 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd7c415d330 0x7fd7c40cc5c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7c40d3730 0x7fd7c41535d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 --2- 172.21.15.46:0/1568939521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7c40d2910 0x7fd7c41f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.756+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 >> 172.21.15.46:0/1568939521 conn(0x7fd7c4072020 msgr2=0x7fd7c40d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.757+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 shutdown_connections 2023-12-16T21:15:56.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.757+0000 7fd7ca3a9640 1 -- 172.21.15.46:0/1568939521 wait complete. 2023-12-16T21:15:56.767 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:15:29.641456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:15:30.976721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:56.767 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:56.876 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.877+0000 7fda42e6e640 1 Processor -- start 2023-12-16T21:15:56.877 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.877+0000 7fda42e6e640 1 -- start start 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda42e6e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c155790 0x7fda3c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda42e6e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda3c06abe0 0x7fda3c15ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda42e6e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3c155b70 con 0x7fda3c15b5e0 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda42e6e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3c06ee30 con 0x7fda3c06abe0 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda42e6e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c06f3b0 con 0x7fda3c155790 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda3c06abe0 0x7fda3c15ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fda3c06abe0 0x7fda3c15ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48040/0 (socket says 172.21.15.46:48040) 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 -- 172.21.15.46:0/3649911531 learned_addr learned my addr 172.21.15.46:0/3649911531 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda40c0a640 1 --2- 172.21.15.46:0/3649911531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c155790 0x7fda3c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 -- 172.21.15.46:0/3649911531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c155790 msgr2=0x7fda3c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:56.878 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 --2- 172.21.15.46:0/3649911531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c155790 0x7fda3c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 -- 172.21.15.46:0/3649911531 >> v1:172.21.15.121:6789/0 conn(0x7fda3c15b5e0 legacy=0x7fda3c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:56.879 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.878+0000 7fda4140b640 1 -- 172.21.15.46:0/3649911531 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3c15c170 con 0x7fda3c06abe0 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda4140b640 1 --2- 172.21.15.46:0/3649911531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c06abe0 0x7fda3c15ba60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fda34009e50 tx=0x7fda3402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d859d43681b6d22 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda3b7fe640 1 -- 172.21.15.46:0/3649911531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3402ca70 con 0x7fda3c06abe0 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda3b7fe640 1 -- 172.21.15.46:0/3649911531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3402cbd0 con 0x7fda3c06abe0 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda3b7fe640 1 -- 172.21.15.46:0/3649911531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda340343f0 con 0x7fda3c06abe0 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda42e6e640 1 -- 172.21.15.46:0/3649911531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c06abe0 msgr2=0x7fda3c15ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda42e6e640 1 --2- 172.21.15.46:0/3649911531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c06abe0 0x7fda3c15ba60 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fda34009e50 tx=0x7fda3402bed0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda42e6e640 1 -- 172.21.15.46:0/3649911531 shutdown_connections 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda42e6e640 1 --2- 172.21.15.46:0/3649911531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c06abe0 0x7fda3c15ba60 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.879+0000 7fda42e6e640 1 --2- 172.21.15.46:0/3649911531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c155790 0x7fda3c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.880+0000 7fda42e6e640 1 -- 172.21.15.46:0/3649911531 >> 172.21.15.46:0/3649911531 conn(0x7fda3c0b1e50 msgr2=0x7fda3c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.880+0000 7fda42e6e640 1 -- 172.21.15.46:0/3649911531 shutdown_connections 2023-12-16T21:15:56.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.880+0000 7fda42e6e640 1 -- 172.21.15.46:0/3649911531 wait complete. 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.881+0000 7fda42e6e640 1 Processor -- start 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.881+0000 7fda42e6e640 1 -- start start 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c06abe0 0x7fda3c1537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c155790 0x7fda3c1e4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 0x7fda3c1e4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c1f0cb0 con 0x7fda3c15b5e0 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c158060 con 0x7fda3c155790 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c158340 con 0x7fda3c06abe0 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 0x7fda3c1e4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 0x7fda3c1e4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48050/0 (socket says 172.21.15.46:48050) 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 -- 172.21.15.46:0/1386018682 learned_addr learned my addr 172.21.15.46:0/1386018682 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda40c0a640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c06abe0 0x7fda3c1537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda3bfff640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c155790 0x7fda3c1e4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.883 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 -- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c06abe0 msgr2=0x7fda3c1537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c06abe0 0x7fda3c1537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 -- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c155790 msgr2=0x7fda3c1e4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c155790 0x7fda3c1e4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 -- 172.21.15.46:0/1386018682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda34009b30 con 0x7fda3c15b5e0 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda3bfff640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c155790 0x7fda3c1e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda4140b640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 0x7fda3c1e4fe0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fda34036040 tx=0x7fda34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda34009510 con 0x7fda3c15b5e0 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3403e450 con 0x7fda3c15b5e0 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.882+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c158620 con 0x7fda3c15b5e0 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.883+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3403e7e0 con 0x7fda3c15b5e0 2023-12-16T21:15:56.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.883+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c157c80 con 0x7fda3c15b5e0 2023-12-16T21:15:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.883+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c1f0e20 con 0x7fda3c15b5e0 2023-12-16T21:15:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.884+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fda3403e940 con 0x7fda3c15b5e0 2023-12-16T21:15:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.884+0000 7fda397fa640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.884+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda340ac330 con 0x7fda3c15b5e0 2023-12-16T21:15:56.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.885+0000 7fda40c0a640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:56.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.885+0000 7fda40c0a640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fda300079a0 tx=0x7fda30008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:56.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:56.889+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fda3c1f0e20 con 0x7fda3c15b5e0 2023-12-16T21:15:57.063 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.064+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda3c1f0e20 con 0x7fda3c15b5e0 2023-12-16T21:15:57.064 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.064+0000 7fda397fa640 1 -- 172.21.15.46:0/1386018682 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda3c1f0e20 con 0x7fda3c15b5e0 2023-12-16T21:15:57.064 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 msgr2=0x7fda2006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fda300079a0 tx=0x7fda30008040 comp rx=0 tx=0).stop 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 msgr2=0x7fda3c1e4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 0x7fda3c1e4fe0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fda34036040 tx=0x7fda34004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 shutdown_connections 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fda2006ae40 0x7fda2006d300 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda3c15b5e0 0x7fda3c1e4fe0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.066 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c155790 0x7fda3c1e4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 --2- 172.21.15.46:0/1386018682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c06abe0 0x7fda3c1537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 >> 172.21.15.46:0/1386018682 conn(0x7fda3c0b1e50 msgr2=0x7fda3c15d890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 shutdown_connections 2023-12-16T21:15:57.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.066+0000 7fda42e6e640 1 -- 172.21.15.46:0/1386018682 wait complete. 2023-12-16T21:15:57.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:57.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.186+0000 7fbce01a9640 1 Processor -- start 2023-12-16T21:15:57.186 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.186+0000 7fbce01a9640 1 -- start start 2023-12-16T21:15:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbce01a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbcd80b5370 0x7fbcd80cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbce01a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd80cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbce01a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd80ce4c0 con 0x7fbcd815d2d0 2023-12-16T21:15:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbce01a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd80d1bb0 con 0x7fbcd80b5370 2023-12-16T21:15:57.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbce01a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd815d6b0 con 0x7fbcd80d2040 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbcdd744640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbcd80b5370 0x7fbcd80cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbcdd744640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fbcd80b5370 0x7fbcd80cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48058/0 (socket says 172.21.15.46:48058) 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbcdd744640 1 -- 172.21.15.46:0/148753214 learned_addr learned my addr 172.21.15.46:0/148753214 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.187+0000 7fbcde746640 1 --2- 172.21.15.46:0/148753214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd80cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.188+0000 7fbcdd744640 1 -- 172.21.15.46:0/148753214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 msgr2=0x7fbcd80cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.188+0000 7fbcdd744640 1 --2- 172.21.15.46:0/148753214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd80cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.188+0000 7fbcdd744640 1 -- 172.21.15.46:0/148753214 >> v1:172.21.15.121:6789/0 conn(0x7fbcd815d2d0 legacy=0x7fbcd80b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.188+0000 7fbcdd744640 1 -- 172.21.15.46:0/148753214 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd80ce700 con 0x7fbcd80b5370 2023-12-16T21:15:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.188+0000 7fbcdd744640 1 --2- 172.21.15.46:0/148753214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd80cda40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009870 tx=0x7fbccc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3de4ec960b3b32 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbcdcf43640 1 -- 172.21.15.46:0/148753214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc030840 con 0x7fbcd80b5370 2023-12-16T21:15:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbcdcf43640 1 -- 172.21.15.46:0/148753214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbccc0309a0 con 0x7fbcd80b5370 2023-12-16T21:15:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbcdcf43640 1 -- 172.21.15.46:0/148753214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbccc030d30 con 0x7fbcd80b5370 2023-12-16T21:15:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 -- 172.21.15.46:0/148753214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 msgr2=0x7fbcd80cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.189 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 --2- 172.21.15.46:0/148753214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd80cda40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009870 tx=0x7fbccc02b980 comp rx=0 tx=0).stop 2023-12-16T21:15:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 -- 172.21.15.46:0/148753214 shutdown_connections 2023-12-16T21:15:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 --2- 172.21.15.46:0/148753214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd80cdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 --2- 172.21.15.46:0/148753214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd80cda40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 -- 172.21.15.46:0/148753214 >> 172.21.15.46:0/148753214 conn(0x7fbcd8072050 msgr2=0x7fbcd8072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 -- 172.21.15.46:0/148753214 shutdown_connections 2023-12-16T21:15:57.190 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.189+0000 7fbce01a9640 1 -- 172.21.15.46:0/148753214 wait complete. 2023-12-16T21:15:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.190+0000 7fbce01a9640 1 Processor -- start 2023-12-16T21:15:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.190+0000 7fbce01a9640 1 -- start start 2023-12-16T21:15:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbce01a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd8153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbce01a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd815ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.196 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbce01a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd815d2d0 0x7fbcd815f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbce01a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81603a0 con 0x7fbcd80b5370 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbce01a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81607c0 con 0x7fbcd815d2d0 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbce01a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd815f720 con 0x7fbcd80d2040 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbcddf45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd8153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbcddf45640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd8153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48072/0 (socket says 172.21.15.46:48072) 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbcddf45640 1 -- 172.21.15.46:0/937186359 learned_addr learned my addr 172.21.15.46:0/937186359 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbcdd744640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd815ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbcde746640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd815d2d0 0x7fbcd815f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.191+0000 7fbcddf45640 1 -- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 msgr2=0x7fbcd815ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcddf45640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd815ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcddf45640 1 -- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd815d2d0 msgr2=0x7fbcd815f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcddf45640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd815d2d0 0x7fbcd815f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcddf45640 1 -- 172.21.15.46:0/937186359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc0094d0 con 0x7fbcd80b5370 2023-12-16T21:15:57.197 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcdd744640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd815ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcde746640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd815d2d0 0x7fbcd815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcddf45640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd8153210 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c400 tx=0x7fbcd000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00052d0 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd0005430 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd815fa00 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0003830 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.192+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd815ffc0 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.193+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd8069400 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.193+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fbcd0003990 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.194+0000 7fbcd67fc640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbcc406ae40 0x7fbcc406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.194+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbcd0088a00 con 0x7fbcd80b5370 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.194+0000 7fbcdd744640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbcc406ae40 0x7fbcc406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.198 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.195+0000 7fbcdd744640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbcc406ae40 0x7fbcc406d300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbccc036000 tx=0x7fbccc03b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.199 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.198+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fbcd0053ab0 con 0x7fbcd80b5370 2023-12-16T21:15:57.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.376+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbcd80ce4c0 con 0x7fbcd80b5370 2023-12-16T21:15:57.376 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.376+0000 7fbcd67fc640 1 -- 172.21.15.46:0/937186359 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbcd0017020 con 0x7fbcd80b5370 2023-12-16T21:15:57.376 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbcc406ae40 msgr2=0x7fbcc406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbcc406ae40 0x7fbcc406d300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fbccc036000 tx=0x7fbccc03b040 comp rx=0 tx=0).stop 2023-12-16T21:15:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 msgr2=0x7fbcd8153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd8153210 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c400 tx=0x7fbcd000c8c0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 shutdown_connections 2023-12-16T21:15:57.378 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fbcc406ae40 0x7fbcc406d300 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd815d2d0 0x7fbcd815f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd80d2040 0x7fbcd815ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 --2- 172.21.15.46:0/937186359 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbcd80b5370 0x7fbcd8153210 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 >> 172.21.15.46:0/937186359 conn(0x7fbcd8072050 msgr2=0x7fbcd80cd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 shutdown_connections 2023-12-16T21:15:57.379 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.378+0000 7fbce01a9640 1 -- 172.21.15.46:0/937186359 wait complete. 2023-12-16T21:15:57.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:57.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.498+0000 7f02be590640 1 Processor -- start 2023-12-16T21:15:57.498 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.498+0000 7f02be590640 1 -- start start 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02be590640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 0x7f02b80cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02be590640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 0x7f02b80b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02be590640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b8071770 con 0x7f02b8144980 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02be590640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b815ced0 con 0x7f02b80ceb70 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02be590640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80b4ce0 con 0x7f02b80b4370 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02bcb2d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 0x7f02b80b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.499 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.499+0000 7f02b77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 0x7f02b80cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02bcb2d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 0x7f02b80b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.46:45228/0 (socket says 172.21.15.46:45228) 2023-12-16T21:15:57.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02bcb2d640 1 -- 172.21.15.46:0/63673649 learned_addr learned my addr 172.21.15.46:0/63673649 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:57.500 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02b77fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 0x7f02b80cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.46:47706/0 (socket says 172.21.15.46:47706) 2023-12-16T21:15:57.501 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02bcb2d640 1 -- 172.21.15.46:0/63673649 >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 msgr2=0x7f02b80cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02bcb2d640 1 --2- 172.21.15.46:0/63673649 >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 0x7f02b80cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02bcb2d640 1 -- 172.21.15.46:0/63673649 >> v1:172.21.15.121:6789/0 conn(0x7f02b8144980 legacy=0x7f02b815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02bcb2d640 1 -- 172.21.15.46:0/63673649 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b80ccdc0 con 0x7f02b80b4370 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02b6ffd640 1 -- 172.21.15.46:0/63673649 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088889834 0 0) 0x7f02b8071770 con 0x7f02b8144980 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.500+0000 7f02b77fe640 1 --2- 172.21.15.46:0/63673649 >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 0x7f02b80cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02bcb2d640 1 --2- 172.21.15.46:0/63673649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 0x7f02b80b47a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009e50 tx=0x7f02b002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=498f45d114257377 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02b6ffd640 1 -- 172.21.15.46:0/63673649 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b002cb00 con 0x7f02b80b4370 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02b6ffd640 1 -- 172.21.15.46:0/63673649 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b002cc60 con 0x7f02b80b4370 2023-12-16T21:15:57.502 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02b6ffd640 1 -- 172.21.15.46:0/63673649 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b00343f0 con 0x7f02b80b4370 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02be590640 1 -- 172.21.15.46:0/63673649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 msgr2=0x7f02b80b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02be590640 1 --2- 172.21.15.46:0/63673649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 0x7f02b80b47a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009e50 tx=0x7f02b002bf60 comp rx=0 tx=0).stop 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02be590640 1 -- 172.21.15.46:0/63673649 shutdown_connections 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02be590640 1 --2- 172.21.15.46:0/63673649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80b4370 0x7f02b80b47a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02be590640 1 --2- 172.21.15.46:0/63673649 >> v2:172.21.15.121:3300/0 conn(0x7f02b80ceb70 0x7f02b80cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.501+0000 7f02be590640 1 -- 172.21.15.46:0/63673649 >> 172.21.15.46:0/63673649 conn(0x7f02b806a4d0 msgr2=0x7f02b806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.502+0000 7f02be590640 1 -- 172.21.15.46:0/63673649 shutdown_connections 2023-12-16T21:15:57.503 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.502+0000 7f02be590640 1 -- 172.21.15.46:0/63673649 wait complete. 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.503+0000 7f02be590640 1 Processor -- start 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.503+0000 7f02be590640 1 -- start start 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02be590640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80b4370 0x7f02b81f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02be590640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80ceb70 0x7f02b80ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02be590640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 0x7f02b8157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02be590640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80cb430 con 0x7f02b8144980 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02be590640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b81552f0 con 0x7f02b80b4370 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02be590640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b80cae70 con 0x7f02b80ceb70 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 0x7f02b8157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.506 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 0x7f02b8157190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48086/0 (socket says 172.21.15.46:48086) 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 -- 172.21.15.46:0/2673851796 learned_addr learned my addr 172.21.15.46:0/2673851796 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02b77fe640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80ceb70 0x7f02b80ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02b7fff640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80b4370 0x7f02b81f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 -- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80ceb70 msgr2=0x7f02b80ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80ceb70 0x7f02b80ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 -- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80b4370 msgr2=0x7f02b81f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80b4370 0x7f02b81f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 -- 172.21.15.46:0/2673851796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8009400 con 0x7f02b8144980 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02b7fff640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80b4370 0x7f02b81f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02b77fe640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80ceb70 0x7f02b80ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.504+0000 7f02bcb2d640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 0x7f02b8157190 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f02b0036040 tx=0x7f02b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.505+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b0009510 con 0x7f02b8144980 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.505+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b003e450 con 0x7f02b8144980 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.505+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b0009b30 con 0x7f02b8144980 2023-12-16T21:15:57.507 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.505+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b003e5e0 con 0x7f02b8144980 2023-12-16T21:15:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.505+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b80c52e0 con 0x7f02b8144980 2023-12-16T21:15:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.506+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b80c5140 con 0x7f02b8144980 2023-12-16T21:15:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.506+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f02b003e740 con 0x7f02b8144980 2023-12-16T21:15:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.507+0000 7f02b4ff9640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f02a406ae40 0x7f02a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.507+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02b00ac120 con 0x7f02b8144980 2023-12-16T21:15:57.508 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.507+0000 7f02b7fff640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f02a406ae40 0x7f02a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.509 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.507+0000 7f02b7fff640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f02a406ae40 0x7f02a406d300 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f02ac000f60 tx=0x7f02ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.510 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.511+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f02b80c5140 con 0x7f02b8144980 2023-12-16T21:15:57.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.687+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f02b80c5140 con 0x7f02b8144980 2023-12-16T21:15:57.687 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.688+0000 7f02b4ff9640 1 -- 172.21.15.46:0/2673851796 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f02b80c5140 con 0x7f02b8144980 2023-12-16T21:15:57.688 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:15:57.689 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f02a406ae40 msgr2=0x7f02a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f02a406ae40 0x7f02a406d300 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f02ac000f60 tx=0x7f02ac009040 comp rx=0 tx=0).stop 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 msgr2=0x7f02b8157190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 0x7f02b8157190 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f02b0036040 tx=0x7f02b0004a40 comp rx=0 tx=0).stop 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 shutdown_connections 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f02a406ae40 0x7f02a406d300 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f02b8144980 0x7f02b8157190 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80ceb70 0x7f02b80ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 --2- 172.21.15.46:0/2673851796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80b4370 0x7f02b81f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 >> 172.21.15.46:0/2673851796 conn(0x7f02b806a4d0 msgr2=0x7f02b8143b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 shutdown_connections 2023-12-16T21:15:57.690 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.690+0000 7f02be590640 1 -- 172.21.15.46:0/2673851796 wait complete. 2023-12-16T21:15:57.700 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-16T21:15:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.809+0000 7fe3f343d640 1 Processor -- start 2023-12-16T21:15:57.809 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.809+0000 7fe3f343d640 1 -- start start 2023-12-16T21:15:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f343d640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe3ec143490 0x7fe3ec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f343d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3ec15e730 0x7fe3ec0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.810 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f343d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ec15e2a0 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f343d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec15b520 con 0x7fe3ec15e730 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f343d640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ec143870 con 0x7fe3ec143490 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3ec15e730 0x7fe3ec0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe3ec15e730 0x7fe3ec0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48102/0 (socket says 172.21.15.46:48102) 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 -- 172.21.15.46:0/4037742265 learned_addr learned my addr 172.21.15.46:0/4037742265 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 -- 172.21.15.46:0/4037742265 >> v2:172.21.15.121:3300/0 conn(0x7fe3ec143490 msgr2=0x7fe3ec0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 --2- 172.21.15.46:0/4037742265 >> v2:172.21.15.121:3300/0 conn(0x7fe3ec143490 0x7fe3ec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 -- 172.21.15.46:0/4037742265 >> v1:172.21.15.121:6789/0 conn(0x7fe3ec15a4e0 legacy=0x7fe3ec15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:57.811 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.810+0000 7fe3f09d8640 1 -- 172.21.15.46:0/4037742265 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ec15b240 con 0x7fe3ec15e730 2023-12-16T21:15:57.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.811+0000 7fe3f09d8640 1 --2- 172.21.15.46:0/4037742265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15e730 0x7fe3ec0b1230 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec06e620 tx=0x7fe3dc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8def1b44e2148e14 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.811+0000 7fe3e3fff640 1 -- 172.21.15.46:0/4037742265 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc02ee90 con 0x7fe3ec15e730 2023-12-16T21:15:57.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.811+0000 7fe3e3fff640 1 -- 172.21.15.46:0/4037742265 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc036440 con 0x7fe3ec15e730 2023-12-16T21:15:57.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.811+0000 7fe3e3fff640 1 -- 172.21.15.46:0/4037742265 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc0367d0 con 0x7fe3ec15e730 2023-12-16T21:15:57.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 -- 172.21.15.46:0/4037742265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15e730 msgr2=0x7fe3ec0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.812 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 --2- 172.21.15.46:0/4037742265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15e730 0x7fe3ec0b1230 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec06e620 tx=0x7fe3dc02e0e0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 -- 172.21.15.46:0/4037742265 shutdown_connections 2023-12-16T21:15:57.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 --2- 172.21.15.46:0/4037742265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15e730 0x7fe3ec0b1230 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 --2- 172.21.15.46:0/4037742265 >> v2:172.21.15.121:3300/0 conn(0x7fe3ec143490 0x7fe3ec0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 -- 172.21.15.46:0/4037742265 >> 172.21.15.46:0/4037742265 conn(0x7fe3ec06a4d0 msgr2=0x7fe3ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:57.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 -- 172.21.15.46:0/4037742265 shutdown_connections 2023-12-16T21:15:57.813 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.812+0000 7fe3f343d640 1 -- 172.21.15.46:0/4037742265 wait complete. 2023-12-16T21:15:57.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.813+0000 7fe3f343d640 1 Processor -- start 2023-12-16T21:15:57.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.813+0000 7fe3f343d640 1 -- start start 2023-12-16T21:15:57.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f343d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ec143490 0x7fe3ec13fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f343d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 0x7fe3ec1eed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.815 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f343d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3ec15e730 0x7fe3ec158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f343d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec0af650 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f343d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec1ef250 con 0x7fe3ec143490 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f343d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ec159370 con 0x7fe3ec15e730 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 0x7fe3ec1eed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 0x7fe3ec1eed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48118/0 (socket says 172.21.15.46:48118) 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 -- 172.21.15.46:0/2283649029 learned_addr learned my addr 172.21.15.46:0/2283649029 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f19da640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3ec15e730 0x7fe3ec158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f11d9640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ec143490 0x7fe3ec13fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 -- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3ec15e730 msgr2=0x7fe3ec158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3ec15e730 0x7fe3ec158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 -- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ec143490 msgr2=0x7fe3ec13fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ec143490 0x7fe3ec13fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.814+0000 7fe3f09d8640 1 -- 172.21.15.46:0/2283649029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc00b770 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.815+0000 7fe3f09d8640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 0x7fe3ec1eed10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc02e120 tx=0x7fe3dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.816 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.815+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc034950 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.815+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3dc034ab0 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.815+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec13ef90 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.815+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3dc034e60 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.815+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ec159da0 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.817 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.816+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ec0af820 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.816+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe3dc03d040 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.817+0000 7fe3e1ffb640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe3d006ae40 0x7fe3d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.817+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3dc0ae150 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.817+0000 7fe3f11d9640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe3d006ae40 0x7fe3d006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:57.818 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.817+0000 7fe3f11d9640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe3d006ae40 0x7fe3d006d300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec067770 tx=0x7fe3e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:57.820 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:57.821+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe3ec0af820 con 0x7fe3ec15a4e0 2023-12-16T21:15:57.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:15:57.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:15:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.012+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe3ec0af820 con 0x7fe3ec15a4e0 2023-12-16T21:15:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.013+0000 7fe3e1ffb640 1 -- 172.21.15.46:0/2283649029 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe3ec0af820 con 0x7fe3ec15a4e0 2023-12-16T21:15:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:listed 0 entries 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe3d006ae40 msgr2=0x7fe3d006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe3d006ae40 0x7fe3d006d300 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec067770 tx=0x7fe3e4008040 comp rx=0 tx=0).stop 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 msgr2=0x7fe3ec1eed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 0x7fe3ec1eed10 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc02e120 tx=0x7fe3dc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 shutdown_connections 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe3d006ae40 0x7fe3d006d300 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3ec15a4e0 0x7fe3ec1eed10 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3ec15e730 0x7fe3ec158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 --2- 172.21.15.46:0/2283649029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3ec143490 0x7fe3ec13fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 >> 172.21.15.46:0/2283649029 conn(0x7fe3ec06a4d0 msgr2=0x7fe3ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 shutdown_connections 2023-12-16T21:15:58.016 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.015+0000 7fe3f343d640 1 -- 172.21.15.46:0/2283649029 wait complete. 2023-12-16T21:15:58.025 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2023-12-16T21:15:58.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:58.026 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-16T21:15:58.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi046.stderr:ceph-fuse[111487]: fuse finished with error 0 and tester_r 0 2023-12-16T21:15:58.124 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:58.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:58.124 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:15:58.217 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:15:58.217 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:58.235 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:15:58.236 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:15:58.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:58.236 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-16T21:15:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:58.294 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-16T21:15:58.294 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:15:58.294 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:58.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:15:58.309 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:15:58.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:58.310 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-16T21:15:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:58.365 INFO:teuthology.orchestra.run.smithi046.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-16T21:15:58.365 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:15:58.365 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:15:58.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:15:58.383 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:15:58.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:15:58.384 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-16T21:15:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:15:58.441 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-16T21:15:58.441 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:15:58.548 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.549+0000 7f4192561640 1 Processor -- start 2023-12-16T21:15:58.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.549+0000 7f4192561640 1 -- start start 2023-12-16T21:15:58.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.549+0000 7f4192561640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f418c15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.549+0000 7f4192561640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c15e2a0 con 0x7f418c143490 2023-12-16T21:15:58.552 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.549+0000 7f4192561640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c15b240 con 0x7f418c0b1400 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.549+0000 7f4192561640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c15b520 con 0x7f418c15e730 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418bfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f418c143490 0x7f418c0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49874/0 (socket says 172.21.15.46:49874) 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418bfff640 1 -- 172.21.15.46:0/1170521850 learned_addr learned my addr 172.21.15.46:0/1170521850 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f4190afe640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f418c15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314487377 0 0) 0x7f418c15e2a0 con 0x7f418c143490 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4174003520 con 0x7f418c143490 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579311973 0 0) 0x7f418c15b240 con 0x7f418c0b1400 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f418c15e2a0 con 0x7f418c0b1400 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916042455 0 0) 0x7f4174003520 con 0x7f418c143490 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 msgr2=0x7f418c15abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f418c15abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 >> v1:172.21.15.121:6789/0 conn(0x7f418c0b1400 legacy=0x7f418c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.553 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c15dc50 con 0x7f418c143490 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f4190afe640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f418c15abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f417c001670 con 0x7f418c143490 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 >> v1:172.21.15.46:6789/0 conn(0x7f418c143490 legacy=0x7f418c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4174003b30 0x7f4174003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f41740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174004d00 0x7f4174005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c15b520 con 0x7f4174003b30 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f417400ba30 con 0x7f4174004d00 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417400bbf0 con 0x7f418c15e730 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.550+0000 7f418bfff640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4174003b30 0x7f4174003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418b7fe640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f41740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f4190afe640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174004d00 0x7f4174005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418bfff640 1 -- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 msgr2=0x7f41740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418bfff640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c15e730 0x7f41740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.554 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418bfff640 1 -- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174004d00 msgr2=0x7f4174005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418bfff640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174004d00 0x7f4174005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418bfff640 1 -- 172.21.15.46:0/1170521850 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4184009a70 con 0x7f4174003b30 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f4190afe640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174004d00 0x7f4174005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.551+0000 7f418bfff640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4174003b30 0x7f4174003f30 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f417c007f90 tx=0x7f41740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5c5a97090e1717 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c005ad0 con 0x7f4174003b30 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c005c30 con 0x7f4174003b30 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f418affd640 1 -- 172.21.15.46:0/1170521850 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c02c790 con 0x7f4174003b30 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4192561640 1 -- 172.21.15.46:0/1170521850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4174003b30 msgr2=0x7f4174003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4192561640 1 --2- 172.21.15.46:0/1170521850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4174003b30 0x7f4174003f30 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f417c007f90 tx=0x7f41740057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4190afe640 1 -- 172.21.15.46:0/1170521850 reap_dead start 2023-12-16T21:15:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4192561640 1 -- 172.21.15.46:0/1170521850 shutdown_connections 2023-12-16T21:15:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4192561640 1 -- 172.21.15.46:0/1170521850 >> 172.21.15.46:0/1170521850 conn(0x7f418c06a4d0 msgr2=0x7f418c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4192561640 1 -- 172.21.15.46:0/1170521850 shutdown_connections 2023-12-16T21:15:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.552+0000 7f4192561640 1 -- 172.21.15.46:0/1170521850 wait complete. 2023-12-16T21:15:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.553+0000 7f4192561640 1 Processor -- start 2023-12-16T21:15:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 -- start start 2023-12-16T21:15:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174003b30 0x7f418c1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 0x7f418c1f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c0a8ee0 0x7f418c0a9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0a98d0 con 0x7f418c142350 2023-12-16T21:15:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c1f1c80 con 0x7f4174003b30 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f4192561640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c1f1f60 con 0x7f418c0a8ee0 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f418b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 0x7f418c1f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f418b7fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 0x7f418c1f13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48142/0 (socket says 172.21.15.46:48142) 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.554+0000 7f418b7fe640 1 -- 172.21.15.46:0/154499473 learned_addr learned my addr 172.21.15.46:0/154499473 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418bfff640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174003b30 0x7f418c1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f4190afe640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c0a8ee0 0x7f418c0a9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418b7fe640 1 -- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c0a8ee0 msgr2=0x7f418c0a9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418b7fe640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c0a8ee0 0x7f418c0a9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418b7fe640 1 -- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174003b30 msgr2=0x7f418c1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418b7fe640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174003b30 0x7f418c1596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418b7fe640 1 -- 172.21.15.46:0/154499473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c1f2240 con 0x7f418c142350 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f418b7fe640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 0x7f418c1f13a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f418000f7a0 tx=0x7f418000fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418000fe50 con 0x7f418c142350 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41800052e0 con 0x7f418c142350 2023-12-16T21:15:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c1f2520 con 0x7f418c142350 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418000d970 con 0x7f418c142350 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.555+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c159c20 con 0x7f418c142350 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.556+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c1f2a40 con 0x7f418c142350 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.557+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f418000dad0 con 0x7f418c142350 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.557+0000 7f4188ff9640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f417806ae70 0x7f417806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.557+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41800894d0 con 0x7f418c142350 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.558+0000 7f418bfff640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f417806ae70 0x7f417806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.558+0000 7f418bfff640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f417806ae70 0x7f417806d330 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f417c034000 tx=0x7f417c009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:58.565 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.562+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f418c1f2a40 con 0x7f418c142350 2023-12-16T21:15:58.773 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.774+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f418c1f2a40 con 0x7f418c142350 2023-12-16T21:15:58.774 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.774+0000 7f4188ff9640 1 -- 172.21.15.46:0/154499473 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4941 (secure 0 0 0) 0x7f418c1f2a40 con 0x7f418c142350 2023-12-16T21:15:58.774 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:58.774 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:15:31.993483+0000","modified":"2023-12-16T21:15:33.986432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"b","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/2015861199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":2015861199},{"type":"v1","addr":"172.21.15.121:6837","nonce":2015861199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-16T21:15:58.775 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f417806ae70 msgr2=0x7f417806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f417806ae70 0x7f417806d330 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f417c034000 tx=0x7f417c009030 comp rx=0 tx=0).stop 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 msgr2=0x7f418c1f13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 0x7f418c1f13a0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f418000f7a0 tx=0x7f418000fc60 comp rx=0 tx=0).stop 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 shutdown_connections 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f417806ae70 0x7f417806d330 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f418c142350 0x7f418c1f13a0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418c0a8ee0 0x7f418c0a9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 --2- 172.21.15.46:0/154499473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4174003b30 0x7f418c1596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 >> 172.21.15.46:0/154499473 conn(0x7f418c06a4d0 msgr2=0x7f418c140410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 shutdown_connections 2023-12-16T21:15:58.777 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.776+0000 7f4192561640 1 -- 172.21.15.46:0/154499473 wait complete. 2023-12-16T21:15:58.787 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:15:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.893+0000 7fba63e1c640 1 Processor -- start 2023-12-16T21:15:58.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba63e1c640 1 -- start start 2023-12-16T21:15:58.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba63e1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c06ea50 0x7fba5c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba63e1c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fba5c160fb0 0x7fba5c161410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba63e1c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c161d20 con 0x7fba5c0b3ff0 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba63e1c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c0d1ee0 con 0x7fba5c160fb0 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba63e1c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c06f370 con 0x7fba5c06ea50 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.894+0000 7fba613b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fba5c160fb0 0x7fba5c161410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.895+0000 7fba613b7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fba5c160fb0 0x7fba5c161410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48158/0 (socket says 172.21.15.46:48158) 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.895+0000 7fba613b7640 1 -- 172.21.15.46:0/48328008 learned_addr learned my addr 172.21.15.46:0/48328008 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.895+0000 7fba613b7640 1 -- 172.21.15.46:0/48328008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c06ea50 msgr2=0x7fba5c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.895+0000 7fba613b7640 1 --2- 172.21.15.46:0/48328008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c06ea50 0x7fba5c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.895+0000 7fba613b7640 1 -- 172.21.15.46:0/48328008 >> v1:172.21.15.46:6789/0 conn(0x7fba5c0b3ff0 legacy=0x7fba5c0b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.895+0000 7fba613b7640 1 -- 172.21.15.46:0/48328008 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c15e760 con 0x7fba5c160fb0 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba613b7640 1 --2- 172.21.15.46:0/48328008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c160fb0 0x7fba5c161410 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fba50009870 tx=0x7fba5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26fb55bfe1b65b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba60bb6640 1 -- 172.21.15.46:0/48328008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50031640 con 0x7fba5c160fb0 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba60bb6640 1 -- 172.21.15.46:0/48328008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba500317a0 con 0x7fba5c160fb0 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba60bb6640 1 -- 172.21.15.46:0/48328008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50031b30 con 0x7fba5c160fb0 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 -- 172.21.15.46:0/48328008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c160fb0 msgr2=0x7fba5c161410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 --2- 172.21.15.46:0/48328008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c160fb0 0x7fba5c161410 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fba50009870 tx=0x7fba5002b8f0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 -- 172.21.15.46:0/48328008 shutdown_connections 2023-12-16T21:15:58.897 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 --2- 172.21.15.46:0/48328008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c160fb0 0x7fba5c161410 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 --2- 172.21.15.46:0/48328008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c06ea50 0x7fba5c06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 -- 172.21.15.46:0/48328008 >> 172.21.15.46:0/48328008 conn(0x7fba5c06a6b0 msgr2=0x7fba5c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 -- 172.21.15.46:0/48328008 shutdown_connections 2023-12-16T21:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.896+0000 7fba63e1c640 1 -- 172.21.15.46:0/48328008 wait complete. 2023-12-16T21:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.897+0000 7fba63e1c640 1 Processor -- start 2023-12-16T21:15:58.898 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.897+0000 7fba63e1c640 1 -- start start 2023-12-16T21:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba63e1c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 0x7fba5c0d0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.900 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba63e1c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c0b3ff0 0x7fba5c13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba63e1c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba5c160fb0 0x7fba5c1faf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba63e1c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1e8cb0 con 0x7fba5c06ea50 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba63e1c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c13f040 con 0x7fba5c160fb0 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba63e1c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1fbdd0 con 0x7fba5c0b3ff0 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba61bb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 0x7fba5c0d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba61bb8640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 0x7fba5c0d0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48160/0 (socket says 172.21.15.46:48160) 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.898+0000 7fba61bb8640 1 -- 172.21.15.46:0/2647879463 learned_addr learned my addr 172.21.15.46:0/2647879463 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba613b7640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c0b3ff0 0x7fba5c13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba623b9640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba5c160fb0 0x7fba5c1faf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba61bb8640 1 -- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c0b3ff0 msgr2=0x7fba5c13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.901 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba61bb8640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c0b3ff0 0x7fba5c13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba61bb8640 1 -- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba5c160fb0 msgr2=0x7fba5c1faf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba61bb8640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba5c160fb0 0x7fba5c1faf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba61bb8640 1 -- 172.21.15.46:0/2647879463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba500094d0 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba61bb8640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 0x7fba5c0d0760 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b410 tx=0x7fba5400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54011830 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba540119b0 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54011d60 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c1e8310 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.899+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c1e88d0 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.900+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c069400 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.900+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fba5401a400 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.901+0000 7fba5a7fc640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba4806ae40 0x7fba4806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.901+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba540897b0 con 0x7fba5c06ea50 2023-12-16T21:15:58.902 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.901+0000 7fba613b7640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba4806ae40 0x7fba4806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:58.903 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.901+0000 7fba613b7640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba4806ae40 0x7fba4806d300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba50036000 tx=0x7fba5003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:58.904 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:58.905+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fba54054860 con 0x7fba5c06ea50 2023-12-16T21:15:59.085 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.086+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba5c067760 con 0x7fba5c06ea50 2023-12-16T21:15:59.086 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.087+0000 7fba5a7fc640 1 -- 172.21.15.46:0/2647879463 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7fba54015090 con 0x7fba5c06ea50 2023-12-16T21:15:59.087 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:15:59.087 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba4806ae40 msgr2=0x7fba4806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba4806ae40 0x7fba4806d300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fba50036000 tx=0x7fba5003c040 comp rx=0 tx=0).stop 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 msgr2=0x7fba5c0d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 0x7fba5c0d0760 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fba5400b410 tx=0x7fba5400b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 shutdown_connections 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fba4806ae40 0x7fba4806d300 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba5c160fb0 0x7fba5c1faf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.089 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba5c0b3ff0 0x7fba5c13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 --2- 172.21.15.46:0/2647879463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba5c06ea50 0x7fba5c0d0760 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 >> 172.21.15.46:0/2647879463 conn(0x7fba5c06a6b0 msgr2=0x7fba5c0b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:59.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 shutdown_connections 2023-12-16T21:15:59.090 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.089+0000 7fba63e1c640 1 -- 172.21.15.46:0/2647879463 wait complete. 2023-12-16T21:15:59.100 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:13:24.010064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-16T21:15:29.641456+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-16T21:15:30.976721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-16T21:15:59.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:15:59.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.210+0000 7f2b76a56640 1 Processor -- start 2023-12-16T21:15:59.210 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.210+0000 7f2b76a56640 1 -- start start 2023-12-16T21:15:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b76a56640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2b70143490 0x7f2b700b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b76a56640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7015e2a0 con 0x7f2b7015e730 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b76a56640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b7015b240 con 0x7f2b7015abf0 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b76a56640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b7015b520 con 0x7f2b70143490 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2b7015e730 0x7f2b7015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49888/0 (socket says 172.21.15.46:49888) 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/2805752973 learned_addr learned my addr 172.21.15.46:0/2805752973 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/2805752973 >> v2:172.21.15.121:3300/0 conn(0x7f2b70143490 0x7f2b700b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1483498546 0 0) 0x7f2b7015e2a0 con 0x7f2b7015e730 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b58003520 con 0x7f2b7015e730 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190299854 0 0) 0x7f2b58003520 con 0x7f2b7015e730 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 >> v2:172.21.15.121:3300/0 conn(0x7f2b70143490 msgr2=0x7f2b700b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 --2- 172.21.15.46:0/2805752973 >> v2:172.21.15.121:3300/0 conn(0x7f2b70143490 0x7f2b700b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 >> v1:172.21.15.121:6789/0 conn(0x7f2b7015abf0 legacy=0x7f2b700b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7015dc50 con 0x7f2b7015e730 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f2b64001670 con 0x7f2b7015e730 2023-12-16T21:15:59.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.211+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 >> v1:172.21.15.46:6789/0 conn(0x7f2b7015e730 legacy=0x7f2b7015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6effd640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b58003970 0x7f2b58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6effd640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6effd640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b58005180 0x7f2b5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7015b520 con 0x7f2b580047c0 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5800e230 con 0x7f2b58003970 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5800e3f0 con 0x7f2b58005180 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b58003970 0x7f2b58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b74ff3640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b58005180 0x7f2b5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b58005180 msgr2=0x7f2b5800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b58005180 0x7f2b5800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b58003970 msgr2=0x7f2b58003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b58003970 0x7f2b58003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b6f7fe640 1 -- 172.21.15.46:0/2805752973 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58003520 con 0x7f2b580047c0 2023-12-16T21:15:59.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b74ff3640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b58005180 0x7f2b5800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.212+0000 7f2b74ff3640 1 -- 172.21.15.46:0/2805752973 reap_dead start 2023-12-16T21:15:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b58004c40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400c240 tx=0x7f2b580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112cc849616fd504 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64005ad0 con 0x7f2b580047c0 2023-12-16T21:15:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b64005c30 con 0x7f2b580047c0 2023-12-16T21:15:59.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b6effd640 1 -- 172.21.15.46:0/2805752973 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6402caf0 con 0x7f2b580047c0 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 -- 172.21.15.46:0/2805752973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 msgr2=0x7f2b58004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b58004c40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400c240 tx=0x7f2b580057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 -- 172.21.15.46:0/2805752973 shutdown_connections 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 --2- 172.21.15.46:0/2805752973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b58004c40 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 -- 172.21.15.46:0/2805752973 >> 172.21.15.46:0/2805752973 conn(0x7f2b7006a4d0 msgr2=0x7f2b7006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 -- 172.21.15.46:0/2805752973 shutdown_connections 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.213+0000 7f2b76a56640 1 -- 172.21.15.46:0/2805752973 wait complete. 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 Processor -- start 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 -- start start 2023-12-16T21:15:59.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b70159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b7013ab90 0x7f2b7013b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b7015b660 0x7f2b7015bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7013cef0 con 0x7f2b580047c0 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7013b550 con 0x7f2b7015b660 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.215+0000 7f2b76a56640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7015c050 con 0x7f2b7013ab90 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b70159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b70159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48182/0 (socket says 172.21.15.46:48182) 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 -- 172.21.15.46:0/4024599340 learned_addr learned my addr 172.21.15.46:0/4024599340 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b74ff3640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b7015b660 0x7f2b7015bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b7013ab90 0x7f2b7013b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 -- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b7013ab90 msgr2=0x7f2b7013b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b7013ab90 0x7f2b7013b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 -- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b7015b660 msgr2=0x7f2b7015bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b7015b660 0x7f2b7015bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 -- 172.21.15.46:0/4024599340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b7015c290 con 0x7f2b580047c0 2023-12-16T21:15:59.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b7013ab90 0x7f2b7013b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6ffff640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b70159740 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2b60011a70 tx=0x7f2b60011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b60009680 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6000fb50 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000fee0 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7015c570 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.216+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7015cb30 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.217+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b70069400 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.218+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2b60014440 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.218+0000 7f2b6cff9640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2b5c06ae40 0x7f2b5c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.218+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b60089f20 con 0x7f2b580047c0 2023-12-16T21:15:59.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.218+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2b5c06ae40 0x7f2b5c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.221 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.219+0000 7f2b6f7fe640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2b5c06ae40 0x7f2b5c06d300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b64034000 tx=0x7f2b64009030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.222 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.222+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2b60054fd0 con 0x7f2b580047c0 2023-12-16T21:15:59.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.397+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b700b2820 con 0x7f2b580047c0 2023-12-16T21:15:59.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.397+0000 7f2b6cff9640 1 -- 172.21.15.46:0/4024599340 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b60022020 con 0x7f2b580047c0 2023-12-16T21:15:59.397 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2023-12-16T21:15:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2b5c06ae40 msgr2=0x7f2b5c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2b5c06ae40 0x7f2b5c06d300 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2b64034000 tx=0x7f2b64009030 comp rx=0 tx=0).stop 2023-12-16T21:15:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 msgr2=0x7f2b70159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b70159740 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2b60011a70 tx=0x7f2b60011f30 comp rx=0 tx=0).stop 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 shutdown_connections 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2b5c06ae40 0x7f2b5c06d300 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b7015b660 0x7f2b7015bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b7013ab90 0x7f2b7013b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 --2- 172.21.15.46:0/4024599340 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2b580047c0 0x7f2b70159740 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 >> 172.21.15.46:0/4024599340 conn(0x7f2b7006a4d0 msgr2=0x7f2b700b0760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 shutdown_connections 2023-12-16T21:15:59.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.399+0000 7f2b76a56640 1 -- 172.21.15.46:0/4024599340 wait complete. 2023-12-16T21:15:59.410 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-16T21:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.519+0000 7f080924f640 1 Processor -- start 2023-12-16T21:15:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.519+0000 7f080924f640 1 -- start start 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080924f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0804144980 0x7f080415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080924f640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f08040ceb70 0x7f08040cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080924f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0804071770 con 0x7f08040b4370 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080924f640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08040ccdc0 con 0x7f08040ceb70 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080924f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cd0a0 con 0x7f0804144980 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080359e640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f08040b4370 0x7f0804071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49890/0 (socket says 172.21.15.46:49890) 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080359e640 1 -- 172.21.15.46:0/4155120461 learned_addr learned my addr 172.21.15.46:0/4155120461 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f0802d9d640 1 --2- 172.21.15.46:0/4155120461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0804144980 0x7f080415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f080259c640 1 --2- 172.21.15.46:0/4155120461 >> v2:172.21.15.121:3300/0 conn(0x7f08040ceb70 0x7f08040cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.520+0000 7f0801d9b640 1 -- 172.21.15.46:0/4155120461 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985457374 0 0) 0x7f0804071770 con 0x7f08040b4370 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f0801d9b640 1 -- 172.21.15.46:0/4155120461 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f0003520 con 0x7f08040b4370 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f0802d9d640 1 -- 172.21.15.46:0/4155120461 >> v2:172.21.15.121:3300/0 conn(0x7f08040ceb70 msgr2=0x7f08040cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f0802d9d640 1 --2- 172.21.15.46:0/4155120461 >> v2:172.21.15.121:3300/0 conn(0x7f08040ceb70 0x7f08040cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f0802d9d640 1 -- 172.21.15.46:0/4155120461 >> v1:172.21.15.46:6789/0 conn(0x7f08040b4370 legacy=0x7f0804071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f0802d9d640 1 -- 172.21.15.46:0/4155120461 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040cd380 con 0x7f0804144980 2023-12-16T21:15:59.522 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f080259c640 1 --2- 172.21.15.46:0/4155120461 >> v2:172.21.15.121:3300/0 conn(0x7f08040ceb70 0x7f08040cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.521+0000 7f0802d9d640 1 --2- 172.21.15.46:0/4155120461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0804144980 0x7f080415c7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0097a0 tx=0x7f07fc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9c91a61fca481ad server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f0801d9b640 1 -- 172.21.15.46:0/4155120461 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc031830 con 0x7f0804144980 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f0801d9b640 1 -- 172.21.15.46:0/4155120461 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc031990 con 0x7f0804144980 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f0801d9b640 1 -- 172.21.15.46:0/4155120461 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc031d00 con 0x7f0804144980 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 -- 172.21.15.46:0/4155120461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0804144980 msgr2=0x7f080415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 --2- 172.21.15.46:0/4155120461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0804144980 0x7f080415c7c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f07fc0097a0 tx=0x7f07fc02b8d0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 -- 172.21.15.46:0/4155120461 shutdown_connections 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 --2- 172.21.15.46:0/4155120461 >> v2:172.21.15.121:3300/0 conn(0x7f08040ceb70 0x7f08040cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 --2- 172.21.15.46:0/4155120461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0804144980 0x7f080415c7c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 -- 172.21.15.46:0/4155120461 >> 172.21.15.46:0/4155120461 conn(0x7f080406a4d0 msgr2=0x7f080406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 -- 172.21.15.46:0/4155120461 shutdown_connections 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.522+0000 7f080924f640 1 -- 172.21.15.46:0/4155120461 wait complete. 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 Processor -- start 2023-12-16T21:15:59.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 -- start start 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 0x7f08040b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08040ceb70 0x7f0804155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804144980 0x7f0804154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040b4130 con 0x7f08040b4370 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040b2200 con 0x7f0804144980 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.524+0000 7f080924f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040b24e0 con 0x7f08040ceb70 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 0x7f08040b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 0x7f08040b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48198/0 (socket says 172.21.15.46:48198) 2023-12-16T21:15:59.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 -- 172.21.15.46:0/1564532405 learned_addr learned my addr 172.21.15.46:0/1564532405 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f080359e640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804144980 0x7f0804154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f080259c640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08040ceb70 0x7f0804155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 -- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08040ceb70 msgr2=0x7f0804155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08040ceb70 0x7f0804155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 -- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804144980 msgr2=0x7f0804154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804144980 0x7f0804154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 -- 172.21.15.46:0/1564532405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec009400 con 0x7f08040b4370 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f080259c640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08040ceb70 0x7f0804155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f0802d9d640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 0x7f08040b3aa0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f07fc02b910 tx=0x7f07fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc004450 con 0x7f08040b4370 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc03f070 con 0x7f08040b4370 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07fc009400 con 0x7f08040b4370 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc003830 con 0x7f08040b4370 2023-12-16T21:15:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.525+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08040b2aa0 con 0x7f08040b4370 2023-12-16T21:15:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.526+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804069400 con 0x7f08040b4370 2023-12-16T21:15:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.527+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f07fc003990 con 0x7f08040b4370 2023-12-16T21:15:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.527+0000 7f07fb7fe640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f07e406ae40 0x7f07e406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.527+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07fc041070 con 0x7f08040b4370 2023-12-16T21:15:59.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.527+0000 7f080259c640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f07e406ae40 0x7f07e406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.528+0000 7f080259c640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f07e406ae40 0x7f07e406d300 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f08040b1c60 tx=0x7f07ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.531+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f07fc076af0 con 0x7f08040b4370 2023-12-16T21:15:59.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.706+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0804143e40 con 0x7f08040b4370 2023-12-16T21:15:59.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.706+0000 7f07fb7fe640 1 -- 172.21.15.46:0/1564532405 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07fc0350a0 con 0x7f08040b4370 2023-12-16T21:15:59.706 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f07e406ae40 msgr2=0x7f07e406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f07e406ae40 0x7f07e406d300 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f08040b1c60 tx=0x7f07ec008040 comp rx=0 tx=0).stop 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 msgr2=0x7f08040b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 0x7f08040b3aa0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f07fc02b910 tx=0x7f07fc0046c0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 shutdown_connections 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f07e406ae40 0x7f07e406d300 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0804144980 0x7f0804154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08040ceb70 0x7f0804155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 --2- 172.21.15.46:0/1564532405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08040b4370 0x7f08040b3aa0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 >> 172.21.15.46:0/1564532405 conn(0x7f080406a4d0 msgr2=0x7f080415dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 shutdown_connections 2023-12-16T21:15:59.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.708+0000 7f080924f640 1 -- 172.21.15.46:0/1564532405 wait complete. 2023-12-16T21:15:59.718 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-16T21:15:59.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.828+0000 7f5fd3a89640 1 Processor -- start 2023-12-16T21:15:59.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.828+0000 7f5fd3a89640 1 -- start start 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.829+0000 7f5fd3a89640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5fcc13cf70 0x7f5fcc071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.829+0000 7f5fd3a89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc15a570 con 0x7f5fcc0721c0 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.829+0000 7f5fd3a89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc15b570 con 0x7f5fcc15e560 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.829+0000 7f5fd3a89640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fcc072670 con 0x7f5fcc13cf70 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.829+0000 7f5fd1024640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5fcc0721c0 0x7f5fcc15de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:49900/0 (socket says 172.21.15.46:49900) 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.829+0000 7f5fd1024640 1 -- 172.21.15.46:0/3984051847 learned_addr learned my addr 172.21.15.46:0/3984051847 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422314710 0 0) 0x7f5fc0001110 con 0x7f5fcc0721c0 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 --2- 172.21.15.46:0/3984051847 >> v2:172.21.15.121:3300/0 conn(0x7f5fcc13cf70 0x7f5fcc071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fb4003520 con 0x7f5fcc0721c0 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926369357 0 0) 0x7f5fb4003520 con 0x7f5fcc0721c0 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 >> v2:172.21.15.121:3300/0 conn(0x7f5fcc13cf70 msgr2=0x7f5fcc071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.831 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 --2- 172.21.15.46:0/3984051847 >> v2:172.21.15.121:3300/0 conn(0x7f5fcc13cf70 0x7f5fcc071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 >> v1:172.21.15.121:6789/0 conn(0x7f5fcc15e560 legacy=0x7f5fcc0cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc15a7b0 con 0x7f5fcc0721c0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f5fc0001800 con 0x7f5fcc0721c0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 >> v1:172.21.15.46:6789/0 conn(0x7f5fcc0721c0 legacy=0x7f5fcc15de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fb4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fb40047c0 0x7f5fb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fb4005180 0x7f5fb400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb400c050 con 0x7f5fb40039c0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb400c1c0 con 0x7f5fb4005180 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb400c380 con 0x7f5fb40047c0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fb4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd2026640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fb4005180 0x7f5fb400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1024640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fb40047c0 0x7f5fb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 -- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fb40047c0 msgr2=0x7f5fb4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fb40047c0 0x7f5fb4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 -- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fb4005180 msgr2=0x7f5fb400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fb4005180 0x7f5fb400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1825640 1 -- 172.21.15.46:0/3984051847 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb4003520 con 0x7f5fb40039c0 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd2026640 1 -- 172.21.15.46:0/3984051847 reap_dead start 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.830+0000 7f5fd1024640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fb40047c0 0x7f5fb4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.831+0000 7f5fd1825640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fb4003de0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009fc0 tx=0x7f5fb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0a1ffc454b7ec9d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.831+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc02cc20 con 0x7f5fb40039c0 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.831+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc00f040 con 0x7f5fb40039c0 2023-12-16T21:15:59.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.831+0000 7f5fd0823640 1 -- 172.21.15.46:0/3984051847 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc034440 con 0x7f5fb40039c0 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 -- 172.21.15.46:0/3984051847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 msgr2=0x7f5fb4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fb4003de0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009fc0 tx=0x7f5fb40057f0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 -- 172.21.15.46:0/3984051847 shutdown_connections 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/3984051847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fb4003de0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 -- 172.21.15.46:0/3984051847 >> 172.21.15.46:0/3984051847 conn(0x7f5fcc06c520 msgr2=0x7f5fcc06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 -- 172.21.15.46:0/3984051847 shutdown_connections 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.832+0000 7f5fd3a89640 1 -- 172.21.15.46:0/3984051847 wait complete. 2023-12-16T21:15:59.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.833+0000 7f5fd3a89640 1 Processor -- start 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.833+0000 7f5fd3a89640 1 -- start start 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd3a89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fcc14aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd3a89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fcc13ca80 0x7f5fcc159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd3a89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fcc1598b0 0x7f5fcc14eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd3a89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc14c950 con 0x7f5fb40039c0 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd3a89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc14c5b0 con 0x7f5fcc13ca80 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd3a89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc14b2e0 con 0x7f5fcc1598b0 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fcc14aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fcc14aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48220/0 (socket says 172.21.15.46:48220) 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 -- 172.21.15.46:0/2947637508 learned_addr learned my addr 172.21.15.46:0/2947637508 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd2026640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fcc1598b0 0x7f5fcc14eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1024640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fcc13ca80 0x7f5fcc159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 -- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fcc1598b0 msgr2=0x7f5fcc14eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fcc1598b0 0x7f5fcc14eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 -- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fcc13ca80 msgr2=0x7f5fcc159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fcc13ca80 0x7f5fcc159370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.834+0000 7f5fd1825640 1 -- 172.21.15.46:0/2947637508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc14b5c0 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.835+0000 7f5fd1825640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fcc14aa60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00e230 tx=0x7f5fbc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.835+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc034c50 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.835+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc034dd0 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.835+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc003bd0 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.835+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc14b8a0 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.835+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc14afa0 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.836+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc069320 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.836+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5fbc040450 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.836+0000 7f5fca7fc640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5fb806ae70 0x7f5fb806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.837+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fbc0abce0 con 0x7f5fb40039c0 2023-12-16T21:15:59.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.837+0000 7f5fd1024640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5fb806ae70 0x7f5fb806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:15:59.839 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.837+0000 7f5fd1024640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5fb806ae70 0x7f5fb806d330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0b0520 tx=0x7f5fc0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:15:59.840 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:15:59.841+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5fbc076d90 con 0x7f5fb40039c0 2023-12-16T21:16:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.015+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5fcc0703e0 con 0x7f5fb40039c0 2023-12-16T21:16:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.015+0000 7f5fca7fc640 1 -- 172.21.15.46:0/2947637508 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5fbc07a3b0 con 0x7f5fb40039c0 2023-12-16T21:16:00.015 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2023-12-16T21:16:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5fb806ae70 msgr2=0x7f5fb806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5fb806ae70 0x7f5fb806d330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc0b0520 tx=0x7f5fc0009040 comp rx=0 tx=0).stop 2023-12-16T21:16:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 msgr2=0x7f5fcc14aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fcc14aa60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc00e230 tx=0x7f5fbc009750 comp rx=0 tx=0).stop 2023-12-16T21:16:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 shutdown_connections 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5fb806ae70 0x7f5fb806d330 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5fcc1598b0 0x7f5fcc14eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5fcc13ca80 0x7f5fcc159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 --2- 172.21.15.46:0/2947637508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5fb40039c0 0x7f5fcc14aa60 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 >> 172.21.15.46:0/2947637508 conn(0x7f5fcc06c520 msgr2=0x7f5fcc1530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 shutdown_connections 2023-12-16T21:16:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.017+0000 7f5fd3a89640 1 -- 172.21.15.46:0/2947637508 wait complete. 2023-12-16T21:16:00.028 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:16:00.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.135+0000 7fc44fcfc640 1 Processor -- start 2023-12-16T21:16:00.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.135+0000 7fc44fcfc640 1 -- start start 2023-12-16T21:16:00.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44fcfc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc448161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44fcfc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4480d0ed0 0x7fc4480d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44fcfc640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc448161d20 con 0x7fc44806ea50 2023-12-16T21:16:00.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44fcfc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44815e760 con 0x7fc4480d0ed0 2023-12-16T21:16:00.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44fcfc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44815d9a0 con 0x7fc448160fb0 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4480d0ed0 0x7fc4480d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fc4480d0ed0 0x7fc4480d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48226/0 (socket says 172.21.15.46:48226) 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 -- 172.21.15.46:0/293302880 learned_addr learned my addr 172.21.15.46:0/293302880 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44d297640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc448161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 -- 172.21.15.46:0/293302880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 msgr2=0x7fc448161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc448161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 -- 172.21.15.46:0/293302880 >> v1:172.21.15.46:6789/0 conn(0x7fc44806ea50 legacy=0x7fc44806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44e299640 1 -- 172.21.15.46:0/293302880 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44815dc80 con 0x7fc4480d0ed0 2023-12-16T21:16:00.137 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.136+0000 7fc44d297640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc448161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.137+0000 7fc44e299640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4480d0ed0 0x7fc4480d1360 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc438009e50 tx=0x7fc43802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e35ee73b49e654d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.137+0000 7fc44ca96640 1 -- 172.21.15.46:0/293302880 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43802cb00 con 0x7fc4480d0ed0 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.137+0000 7fc44ca96640 1 -- 172.21.15.46:0/293302880 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43802cc60 con 0x7fc4480d0ed0 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.137+0000 7fc44ca96640 1 -- 172.21.15.46:0/293302880 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380343f0 con 0x7fc4480d0ed0 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 -- 172.21.15.46:0/293302880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4480d0ed0 msgr2=0x7fc4480d1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4480d0ed0 0x7fc4480d1360 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc438009e50 tx=0x7fc43802bf60 comp rx=0 tx=0).stop 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 -- 172.21.15.46:0/293302880 shutdown_connections 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc4480d0ed0 0x7fc4480d1360 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/293302880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc448161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.139 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 -- 172.21.15.46:0/293302880 >> 172.21.15.46:0/293302880 conn(0x7fc44806a6b0 msgr2=0x7fc44806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 -- 172.21.15.46:0/293302880 shutdown_connections 2023-12-16T21:16:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.138+0000 7fc44fcfc640 1 -- 172.21.15.46:0/293302880 wait complete. 2023-12-16T21:16:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.139+0000 7fc44fcfc640 1 Processor -- start 2023-12-16T21:16:00.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.139+0000 7fc44fcfc640 1 -- start start 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44fcfc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 0x7fc4481568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44fcfc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480d0ed0 0x7fc448157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44fcfc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc44815f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44fcfc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc448158910 con 0x7fc44806ea50 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44fcfc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc448158bf0 con 0x7fc4480d0ed0 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44fcfc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc448158ed0 con 0x7fc448160fb0 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 0x7fc4481568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 0x7fc4481568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48240/0 (socket says 172.21.15.46:48240) 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 -- 172.21.15.46:0/1212590938 learned_addr learned my addr 172.21.15.46:0/1212590938 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:00.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 -- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 msgr2=0x7fc44815f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc44815f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 -- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480d0ed0 msgr2=0x7fc448157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480d0ed0 0x7fc448157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.140+0000 7fc44da98640 1 -- 172.21.15.46:0/1212590938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438009b30 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.141+0000 7fc44da98640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 0x7fc4481568e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc44000c400 tx=0x7fc44000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.141+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4400052d0 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.141+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc440005430 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.141+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4481591b0 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.141+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440003830 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.141+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4481e7840 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.142+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448069400 con 0x7fc44806ea50 2023-12-16T21:16:00.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.142+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fc440003990 con 0x7fc44806ea50 2023-12-16T21:16:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.143+0000 7fc4467fc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc43406ae40 0x7fc43406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.143+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc440088c70 con 0x7fc44806ea50 2023-12-16T21:16:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.143+0000 7fc44d297640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc43406ae40 0x7fc43406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.143+0000 7fc44d297640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc43406ae40 0x7fc43406d300 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc43c006fd0 tx=0x7fc43c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.147+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fc440053c80 con 0x7fc44806ea50 2023-12-16T21:16:00.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.360+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc44806d550 con 0x7fc44806ea50 2023-12-16T21:16:00.360 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.360+0000 7fc4467fc640 1 -- 172.21.15.46:0/1212590938 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fc440017090 con 0x7fc44806ea50 2023-12-16T21:16:00.360 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:00.360 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:15:31.993483+0000","modified":"2023-12-16T21:15:33.986432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"b","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/2015861199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":2015861199},{"type":"v1","addr":"172.21.15.121:6837","nonce":2015861199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-16T21:16:00.360 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-12-16T21:16:00.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc43406ae40 msgr2=0x7fc43406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc43406ae40 0x7fc43406d300 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc43c006fd0 tx=0x7fc43c008040 comp rx=0 tx=0).stop 2023-12-16T21:16:00.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 msgr2=0x7fc4481568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 0x7fc4481568e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc44000c400 tx=0x7fc44000c8c0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 shutdown_connections 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fc43406ae40 0x7fc43406d300 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc448160fb0 0x7fc44815f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4480d0ed0 0x7fc448157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 --2- 172.21.15.46:0/1212590938 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc44806ea50 0x7fc4481568e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.362+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 >> 172.21.15.46:0/1212590938 conn(0x7fc44806a6b0 msgr2=0x7fc44806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.363+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 shutdown_connections 2023-12-16T21:16:00.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.363+0000 7fc44fcfc640 1 -- 172.21.15.46:0/1212590938 wait complete. 2023-12-16T21:16:00.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:16:00.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.479+0000 7f817192c640 1 Processor -- start 2023-12-16T21:16:00.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.479+0000 7f817192c640 1 -- start start 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f817192c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f816c06abe0 0x7f816c15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f817192c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c155b70 con 0x7f816c155790 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f817192c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c06f3b0 con 0x7f816c15b5e0 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f817192c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c06ee30 con 0x7f816c06abe0 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f816b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f816c06abe0 0x7f816c15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f816b7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f816c06abe0 0x7f816c15ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48246/0 (socket says 172.21.15.46:48246) 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.480+0000 7f816b7fe640 1 -- 172.21.15.46:0/2688192788 learned_addr learned my addr 172.21.15.46:0/2688192788 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.481+0000 7f816b7fe640 1 -- 172.21.15.46:0/2688192788 >> v1:172.21.15.121:6789/0 conn(0x7f816c15b5e0 legacy=0x7f816c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.481+0000 7f816b7fe640 1 -- 172.21.15.46:0/2688192788 >> v1:172.21.15.46:6789/0 conn(0x7f816c155790 legacy=0x7f816c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.481+0000 7f816b7fe640 1 -- 172.21.15.46:0/2688192788 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c15c170 con 0x7f816c06abe0 2023-12-16T21:16:00.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f816b7fe640 1 --2- 172.21.15.46:0/2688192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c15ba90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8164009e50 tx=0x7f816402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61dd99942fa6cd69 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f8169ffb640 1 -- 172.21.15.46:0/2688192788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816402cbb0 con 0x7f816c06abe0 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f8169ffb640 1 -- 172.21.15.46:0/2688192788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816402cd10 con 0x7f816c06abe0 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f8169ffb640 1 -- 172.21.15.46:0/2688192788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164034610 con 0x7f816c06abe0 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 -- 172.21.15.46:0/2688192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 msgr2=0x7f816c15ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 --2- 172.21.15.46:0/2688192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c15ba90 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8164009e50 tx=0x7f816402c010 comp rx=0 tx=0).stop 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 -- 172.21.15.46:0/2688192788 shutdown_connections 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 --2- 172.21.15.46:0/2688192788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c15ba90 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 -- 172.21.15.46:0/2688192788 >> 172.21.15.46:0/2688192788 conn(0x7f816c0b1e50 msgr2=0x7f816c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 -- 172.21.15.46:0/2688192788 shutdown_connections 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.482+0000 7f817192c640 1 -- 172.21.15.46:0/2688192788 wait complete. 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.483+0000 7f817192c640 1 Processor -- start 2023-12-16T21:16:00.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 -- start start 2023-12-16T21:16:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f816c155790 0x7f816c14f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f816c15b5e0 0x7f816c0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0b3a00 con 0x7f816c06abe0 2023-12-16T21:16:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c0b36d0 con 0x7f816c155790 2023-12-16T21:16:00.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f817192c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c15a4c0 con 0x7f816c15b5e0 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f816affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f816affd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48248/0 (socket says 172.21.15.46:48248) 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.484+0000 7f816affd640 1 -- 172.21.15.46:0/259051263 learned_addr learned my addr 172.21.15.46:0/259051263 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816a7fc640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f816c155790 0x7f816c14f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816b7fe640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f816c15b5e0 0x7f816c0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816affd640 1 -- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f816c15b5e0 msgr2=0x7f816c0c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816affd640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f816c15b5e0 0x7f816c0c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816affd640 1 -- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f816c155790 msgr2=0x7f816c14f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816affd640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f816c155790 0x7f816c14f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816affd640 1 -- 172.21.15.46:0/259051263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164009b30 con 0x7f816c06abe0 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816a7fc640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f816c155790 0x7f816c14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:00.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f816affd640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c159c70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f815c00c450 tx=0x7f815c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815c0052d0 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815c005430 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c15a7a0 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f815c012870 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.485+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c15a1b0 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.486+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c071480 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.487+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f815c0129d0 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.487+0000 7f8153fff640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f815806ae40 0x7f815806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.487+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f815c089030 con 0x7f816c06abe0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.487+0000 7f816a7fc640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f815806ae40 0x7f815806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.488+0000 7f816a7fc640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f815806ae40 0x7f815806d300 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8160007fd0 tx=0x7f8160009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.491+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f816c071480 con 0x7f816c06abe0 2023-12-16T21:16:00.669 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.669+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f816c071480 con 0x7f816c06abe0 2023-12-16T21:16:00.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.671+0000 7f8153fff640 1 -- 172.21.15.46:0/259051263 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f816c071480 con 0x7f816c06abe0 2023-12-16T21:16:00.670 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:00.671 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f815806ae40 msgr2=0x7f815806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f815806ae40 0x7f815806d300 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f8160007fd0 tx=0x7f8160009040 comp rx=0 tx=0).stop 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 msgr2=0x7f816c159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c159c70 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f815c00c450 tx=0x7f815c00c910 comp rx=0 tx=0).stop 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 shutdown_connections 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f815806ae40 0x7f815806d300 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f816c15b5e0 0x7f816c0c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f816c155790 0x7f816c14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 --2- 172.21.15.46:0/259051263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f816c06abe0 0x7f816c159c70 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 >> 172.21.15.46:0/259051263 conn(0x7f816c0b1e50 msgr2=0x7f816c153110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:00.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 shutdown_connections 2023-12-16T21:16:00.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.673+0000 7f817192c640 1 -- 172.21.15.46:0/259051263 wait complete. 2023-12-16T21:16:00.686 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-16T21:16:00.686 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:16:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.793+0000 7fd6977ec640 1 Processor -- start 2023-12-16T21:16:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.793+0000 7fd6977ec640 1 -- start start 2023-12-16T21:16:00.794 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.794+0000 7fd6977ec640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd69006ea50 0x7fd69006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.794+0000 7fd6977ec640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd690161d20 con 0x7fd690160fb0 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.794+0000 7fd6977ec640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69006f370 con 0x7fd6900d0ed0 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.794+0000 7fd6977ec640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69015e760 con 0x7fd69006ea50 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.794+0000 7fd695588640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd69006ea50 0x7fd69006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.795+0000 7fd695588640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fd69006ea50 0x7fd69006ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58494/0 (socket says 172.21.15.46:58494) 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.795+0000 7fd695588640 1 -- 172.21.15.46:0/1945770078 learned_addr learned my addr 172.21.15.46:0/1945770078 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.795+0000 7fd695588640 1 -- 172.21.15.46:0/1945770078 >> v1:172.21.15.121:6789/0 conn(0x7fd6900d0ed0 legacy=0x7fd6900d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.795+0000 7fd695588640 1 -- 172.21.15.46:0/1945770078 >> v1:172.21.15.46:6789/0 conn(0x7fd690160fb0 legacy=0x7fd690161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.795+0000 7fd68ffff640 1 -- 172.21.15.46:0/1945770078 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63537975 0 0) 0x7fd690161d20 con 0x7fd690160fb0 2023-12-16T21:16:00.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.795+0000 7fd695588640 1 -- 172.21.15.46:0/1945770078 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69015d9a0 con 0x7fd69006ea50 2023-12-16T21:16:00.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd695588640 1 --2- 172.21.15.46:0/1945770078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd69006ea50 0x7fd69006ee30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd688009870 tx=0x7fd68802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409f440a2ddc11c6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd68ffff640 1 -- 172.21.15.46:0/1945770078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688031870 con 0x7fd69006ea50 2023-12-16T21:16:00.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd68ffff640 1 -- 172.21.15.46:0/1945770078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6880319d0 con 0x7fd69006ea50 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd68ffff640 1 -- 172.21.15.46:0/1945770078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688031d60 con 0x7fd69006ea50 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 -- 172.21.15.46:0/1945770078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd69006ea50 msgr2=0x7fd69006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 --2- 172.21.15.46:0/1945770078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd69006ea50 0x7fd69006ee30 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd688009870 tx=0x7fd68802b910 comp rx=0 tx=0).stop 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 -- 172.21.15.46:0/1945770078 shutdown_connections 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 --2- 172.21.15.46:0/1945770078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd69006ea50 0x7fd69006ee30 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 -- 172.21.15.46:0/1945770078 >> 172.21.15.46:0/1945770078 conn(0x7fd69006a6b0 msgr2=0x7fd69006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 -- 172.21.15.46:0/1945770078 shutdown_connections 2023-12-16T21:16:00.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.796+0000 7fd6977ec640 1 -- 172.21.15.46:0/1945770078 wait complete. 2023-12-16T21:16:00.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 Processor -- start 2023-12-16T21:16:00.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 -- start start 2023-12-16T21:16:00.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd69006ea50 0x7fd69014c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6900d0ed0 0x7fd69013da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 0x7fd69013dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69014a830 con 0x7fd690160fb0 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd690149c20 con 0x7fd69006ea50 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd6977ec640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd690149f00 con 0x7fd6900d0ed0 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd695d89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 0x7fd69013dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd695d89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 0x7fd69013dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58500/0 (socket says 172.21.15.46:58500) 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.798+0000 7fd695d89640 1 -- 172.21.15.46:0/367247632 learned_addr learned my addr 172.21.15.46:0/367247632 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695588640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd69006ea50 0x7fd69014c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd694d87640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6900d0ed0 0x7fd69013da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695d89640 1 -- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6900d0ed0 msgr2=0x7fd69013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695d89640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6900d0ed0 0x7fd69013da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695d89640 1 -- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd69006ea50 msgr2=0x7fd69014c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695d89640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd69006ea50 0x7fd69014c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695d89640 1 -- 172.21.15.46:0/367247632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6880094d0 con 0x7fd690160fb0 2023-12-16T21:16:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd695d89640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 0x7fd69013dfd0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd68000c780 tx=0x7fd68000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680004b10 con 0x7fd690160fb0 2023-12-16T21:16:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd680004c70 con 0x7fd690160fb0 2023-12-16T21:16:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69014a1e0 con 0x7fd690160fb0 2023-12-16T21:16:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68000aa90 con 0x7fd690160fb0 2023-12-16T21:16:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.799+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69014cb70 con 0x7fd690160fb0 2023-12-16T21:16:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.800+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69014a9e0 con 0x7fd690160fb0 2023-12-16T21:16:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.801+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fd68000abf0 con 0x7fd690160fb0 2023-12-16T21:16:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.801+0000 7fd68dffb640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd67c06ae40 0x7fd67c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.802+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6800896a0 con 0x7fd690160fb0 2023-12-16T21:16:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.802+0000 7fd695588640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd67c06ae40 0x7fd67c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.802+0000 7fd695588640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd67c06ae40 0x7fd67c06d300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd688036000 tx=0x7fd68803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:00.806+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fd69014a9e0 con 0x7fd690160fb0 2023-12-16T21:16:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.019+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd690069400 con 0x7fd690160fb0 2023-12-16T21:16:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.019+0000 7fd68dffb640 1 -- 172.21.15.46:0/367247632 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4941 (secure 0 0 0) 0x7fd680054760 con 0x7fd690160fb0 2023-12-16T21:16:01.019 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:01.019 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:15:31.993483+0000","modified":"2023-12-16T21:15:33.986432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"b","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/2015861199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":2015861199},{"type":"v1","addr":"172.21.15.121:6837","nonce":2015861199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-16T21:16:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-12-16T21:16:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd67c06ae40 msgr2=0x7fd67c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd67c06ae40 0x7fd67c06d300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd688036000 tx=0x7fd68803c040 comp rx=0 tx=0).stop 2023-12-16T21:16:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 msgr2=0x7fd69013dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 0x7fd69013dfd0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd68000c780 tx=0x7fd68000cc40 comp rx=0 tx=0).stop 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 shutdown_connections 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fd67c06ae40 0x7fd67c06d300 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd690160fb0 0x7fd69013dfd0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6900d0ed0 0x7fd69013da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 --2- 172.21.15.46:0/367247632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd69006ea50 0x7fd69014c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 >> 172.21.15.46:0/367247632 conn(0x7fd69006a6b0 msgr2=0x7fd69015b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 shutdown_connections 2023-12-16T21:16:01.022 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.021+0000 7fd6977ec640 1 -- 172.21.15.46:0/367247632 wait complete. 2023-12-16T21:16:01.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:16:01.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.140+0000 7eff4bba9640 1 Processor -- start 2023-12-16T21:16:01.140 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.140+0000 7eff4bba9640 1 -- start start 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff4bba9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff4415d1d0 0x7eff4406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff4bba9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff440ad640 0x7eff4406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff4bba9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff440adae0 con 0x7eff440aeb80 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff4bba9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff440addc0 con 0x7eff4415d1d0 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff4bba9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff440ae0a0 con 0x7eff440ad640 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff4415d1d0 0x7eff4406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7eff4415d1d0 0x7eff4406b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58516/0 (socket says 172.21.15.46:58516) 2023-12-16T21:16:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 -- 172.21.15.46:0/3313687365 learned_addr learned my addr 172.21.15.46:0/3313687365 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49144640 1 --2- 172.21.15.46:0/3313687365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff440ad640 0x7eff4406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 -- 172.21.15.46:0/3313687365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff440ad640 msgr2=0x7eff4406ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 --2- 172.21.15.46:0/3313687365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff440ad640 0x7eff4406ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 -- 172.21.15.46:0/3313687365 >> v1:172.21.15.121:6789/0 conn(0x7eff440aeb80 legacy=0x7eff44160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.141+0000 7eff49945640 1 -- 172.21.15.46:0/3313687365 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff440ae380 con 0x7eff4415d1d0 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.142+0000 7eff49945640 1 --2- 172.21.15.46:0/3313687365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff4415d1d0 0x7eff4406b4e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7eff34009e50 tx=0x7eff3402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1dfc71b8dd23c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.142+0000 7eff48943640 1 -- 172.21.15.46:0/3313687365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3402ca70 con 0x7eff4415d1d0 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.142+0000 7eff48943640 1 -- 172.21.15.46:0/3313687365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3402cbd0 con 0x7eff4415d1d0 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.142+0000 7eff48943640 1 -- 172.21.15.46:0/3313687365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff340343f0 con 0x7eff4415d1d0 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 -- 172.21.15.46:0/3313687365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff4415d1d0 msgr2=0x7eff4406b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 --2- 172.21.15.46:0/3313687365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff4415d1d0 0x7eff4406b4e0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7eff34009e50 tx=0x7eff3402bed0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.143 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 -- 172.21.15.46:0/3313687365 shutdown_connections 2023-12-16T21:16:01.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 --2- 172.21.15.46:0/3313687365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff440ad640 0x7eff4406ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 --2- 172.21.15.46:0/3313687365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff4415d1d0 0x7eff4406b4e0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 -- 172.21.15.46:0/3313687365 >> 172.21.15.46:0/3313687365 conn(0x7eff44070e70 msgr2=0x7eff44071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:01.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 -- 172.21.15.46:0/3313687365 shutdown_connections 2023-12-16T21:16:01.144 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.143+0000 7eff4bba9640 1 -- 172.21.15.46:0/3313687365 wait complete. 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.144+0000 7eff4bba9640 1 Processor -- start 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.144+0000 7eff4bba9640 1 -- start start 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff4bba9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 0x7eff441500a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff4bba9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff440aeb80 0x7eff4415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff4bba9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff4415d1d0 0x7eff4414d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff4bba9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff440cc4f0 con 0x7eff440ad640 2023-12-16T21:16:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff4bba9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff440cc770 con 0x7eff440aeb80 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff4bba9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff440cca50 con 0x7eff4415d1d0 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 0x7eff441500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 0x7eff441500a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58530/0 (socket says 172.21.15.46:58530) 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 -- 172.21.15.46:0/2280658528 learned_addr learned my addr 172.21.15.46:0/2280658528 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 -- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff4415d1d0 msgr2=0x7eff4414d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff4415d1d0 0x7eff4414d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 -- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff440aeb80 msgr2=0x7eff4415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff440aeb80 0x7eff4415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 -- 172.21.15.46:0/2280658528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34009b30 con 0x7eff440ad640 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.145+0000 7eff49945640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 0x7eff441500a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7eff34009e50 tx=0x7eff34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.146+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3403e450 con 0x7eff440ad640 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.146+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3403e5f0 con 0x7eff440ad640 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.146+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3403e9a0 con 0x7eff440ad640 2023-12-16T21:16:01.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.146+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff440ccd30 con 0x7eff440ad640 2023-12-16T21:16:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.146+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff441505e0 con 0x7eff440ad640 2023-12-16T21:16:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.147+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff44150770 con 0x7eff440ad640 2023-12-16T21:16:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.147+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7eff3403eb00 con 0x7eff440ad640 2023-12-16T21:16:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.147+0000 7eff427fc640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7eff3006ae40 0x7eff3006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.147+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff340ac6a0 con 0x7eff440ad640 2023-12-16T21:16:01.148 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.148+0000 7eff49144640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7eff3006ae40 0x7eff3006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.148+0000 7eff49144640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7eff3006ae40 0x7eff3006d300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff38006fd0 tx=0x7eff38008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.152+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7eff34077870 con 0x7eff440ad640 2023-12-16T21:16:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.362+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff44069400 con 0x7eff440ad640 2023-12-16T21:16:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.363+0000 7eff427fc640 1 -- 172.21.15.46:0/2280658528 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+4941 (secure 0 0 0) 0x7eff3403edb0 con 0x7eff440ad640 2023-12-16T21:16:01.363 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:01.363 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-16T21:15:31.993483+0000","modified":"2023-12-16T21:15:33.986432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5946},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5946":{"gid":5946,"name":"b","rank":0,"incarnation":18,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/2015861199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":2015861199},{"type":"v1","addr":"172.21.15.121:6837","nonce":2015861199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-16T21:16:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 20 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7eff3006ae40 msgr2=0x7eff3006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7eff3006ae40 0x7eff3006d300 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff38006fd0 tx=0x7eff38008040 comp rx=0 tx=0).stop 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 msgr2=0x7eff441500a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 0x7eff441500a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7eff34009e50 tx=0x7eff34004a40 comp rx=0 tx=0).stop 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 shutdown_connections 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7eff3006ae40 0x7eff3006d300 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7eff4415d1d0 0x7eff4414d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7eff440aeb80 0x7eff4415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 --2- 172.21.15.46:0/2280658528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff440ad640 0x7eff441500a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 >> 172.21.15.46:0/2280658528 conn(0x7eff44070e70 msgr2=0x7eff4406a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 shutdown_connections 2023-12-16T21:16:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.365+0000 7eff4bba9640 1 -- 172.21.15.46:0/2280658528 wait complete. 2023-12-16T21:16:01.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:16:01.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.482+0000 7f089c9c9640 1 Processor -- start 2023-12-16T21:16:01.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.482+0000 7f089c9c9640 1 -- start start 2023-12-16T21:16:01.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f089c9c9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0898144980 0x7f089815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f089c9c9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08980ceb70 0x7f08980cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f089c9c9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0898071770 con 0x7f08980b4370 2023-12-16T21:16:01.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f089c9c9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08980ccdc0 con 0x7f08980ceb70 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f089c9c9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08980cd0a0 con 0x7f0898144980 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f0895d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08980ceb70 0x7f08980cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f0895d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f08980ceb70 0x7f08980cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58540/0 (socket says 172.21.15.46:58540) 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f0895d9b640 1 -- 172.21.15.46:0/4096770987 learned_addr learned my addr 172.21.15.46:0/4096770987 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f089659c640 1 --2- 172.21.15.46:0/4096770987 >> v2:172.21.15.121:3300/0 conn(0x7f0898144980 0x7f089815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f0895d9b640 1 -- 172.21.15.46:0/4096770987 >> v2:172.21.15.121:3300/0 conn(0x7f0898144980 msgr2=0x7f089815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f0895d9b640 1 --2- 172.21.15.46:0/4096770987 >> v2:172.21.15.121:3300/0 conn(0x7f0898144980 0x7f089815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.483+0000 7f0895d9b640 1 -- 172.21.15.46:0/4096770987 >> v1:172.21.15.121:6789/0 conn(0x7f08980b4370 legacy=0x7f0898071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:01.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.484+0000 7f0895d9b640 1 -- 172.21.15.46:0/4096770987 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08980cd380 con 0x7f08980ceb70 2023-12-16T21:16:01.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.484+0000 7f0895d9b640 1 --2- 172.21.15.46:0/4096770987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f08980cc450 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f08980b4ec0 tx=0x7f088c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b2cebf1b7bb229c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.484+0000 7f089559a640 1 -- 172.21.15.46:0/4096770987 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c031870 con 0x7f08980ceb70 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.484+0000 7f089559a640 1 -- 172.21.15.46:0/4096770987 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c0319d0 con 0x7f08980ceb70 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.484+0000 7f089559a640 1 -- 172.21.15.46:0/4096770987 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c031d60 con 0x7f08980ceb70 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 -- 172.21.15.46:0/4096770987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 msgr2=0x7f08980cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 --2- 172.21.15.46:0/4096770987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f08980cc450 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f08980b4ec0 tx=0x7f088c02b910 comp rx=0 tx=0).stop 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 -- 172.21.15.46:0/4096770987 shutdown_connections 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 --2- 172.21.15.46:0/4096770987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f08980cc450 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 --2- 172.21.15.46:0/4096770987 >> v2:172.21.15.121:3300/0 conn(0x7f0898144980 0x7f089815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 -- 172.21.15.46:0/4096770987 >> 172.21.15.46:0/4096770987 conn(0x7f089806a4d0 msgr2=0x7f089806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 -- 172.21.15.46:0/4096770987 shutdown_connections 2023-12-16T21:16:01.486 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.485+0000 7f089c9c9640 1 -- 172.21.15.46:0/4096770987 wait complete. 2023-12-16T21:16:01.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.486+0000 7f089c9c9640 1 Processor -- start 2023-12-16T21:16:01.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.486+0000 7f089c9c9640 1 -- start start 2023-12-16T21:16:01.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089c9c9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08980b4370 0x7f0898157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089c9c9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f089813bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089c9c9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0898144980 0x7f0898140050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089c9c9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898140fb0 con 0x7f08980ceb70 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089c9c9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898141ba0 con 0x7f0898144980 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089c9c9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f089813c480 con 0x7f08980b4370 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f0895d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f089813bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f0895d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f089813bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58556/0 (socket says 172.21.15.46:58556) 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f0895d9b640 1 -- 172.21.15.46:0/2796371662 learned_addr learned my addr 172.21.15.46:0/2796371662 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f089659c640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08980b4370 0x7f0898157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.487+0000 7f0896d9d640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0898144980 0x7f0898140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0895d9b640 1 -- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08980b4370 msgr2=0x7f0898157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0895d9b640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08980b4370 0x7f0898157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0895d9b640 1 -- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0898144980 msgr2=0x7f0898140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.489 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0895d9b640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0898144980 0x7f0898140050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0895d9b640 1 -- 172.21.15.46:0/2796371662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c0094d0 con 0x7f08980ceb70 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0895d9b640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f089813bf40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f088c02b950 tx=0x7f088c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c03abe0 con 0x7f08980ceb70 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c03ad40 con 0x7f08980ceb70 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c044610 con 0x7f08980ceb70 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0898141200 con 0x7f08980ceb70 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.488+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08981417c0 con 0x7f08980ceb70 2023-12-16T21:16:01.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.489+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0898069400 con 0x7f08980ceb70 2023-12-16T21:16:01.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.489+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f088c044770 con 0x7f08980ceb70 2023-12-16T21:16:01.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.490+0000 7f0886ffd640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f087806ae40 0x7f087806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.490+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f088c02d040 con 0x7f08980ceb70 2023-12-16T21:16:01.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.490+0000 7f089659c640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f087806ae40 0x7f087806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.491+0000 7f089659c640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f087806ae40 0x7f087806d300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f08900079a0 tx=0x7f0890008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.494+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f088c076e60 con 0x7f08980ceb70 2023-12-16T21:16:01.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.673+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f089815bbf0 con 0x7f08980ceb70 2023-12-16T21:16:01.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.674+0000 7f0886ffd640 1 -- 172.21.15.46:0/2796371662 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f088c035020 con 0x7f08980ceb70 2023-12-16T21:16:01.674 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:01.674 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":50,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:15:33.636677+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-16T21:15:29.641456+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-16T21:15:30.976721+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f087806ae40 msgr2=0x7f087806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f087806ae40 0x7f087806d300 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f08900079a0 tx=0x7f0890008040 comp rx=0 tx=0).stop 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 msgr2=0x7f089813bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f089813bf40 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f088c02b950 tx=0x7f088c0046c0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 shutdown_connections 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f087806ae40 0x7f087806d300 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f08980ceb70 0x7f089813bf40 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0898144980 0x7f0898140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 --2- 172.21.15.46:0/2796371662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08980b4370 0x7f0898157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.676 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 >> 172.21.15.46:0/2796371662 conn(0x7f089806a4d0 msgr2=0x7f08980b21d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 shutdown_connections 2023-12-16T21:16:01.677 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.676+0000 7f089c9c9640 1 -- 172.21.15.46:0/2796371662 wait complete. 2023-12-16T21:16:01.687 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-16T21:16:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.795+0000 7f63f5889640 1 Processor -- start 2023-12-16T21:16:01.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.795+0000 7f63f5889640 1 -- start start 2023-12-16T21:16:01.796 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63f5889640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d6c0 0x7f63f006b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63f5889640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f63f00d1d50 0x7f63f006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63f5889640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f006ce00 con 0x7f63f015d2e0 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63f5889640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f006bf40 con 0x7f63f00d1d50 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63f5889640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f006c1f0 con 0x7f63f015d6c0 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63ee7fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d6c0 0x7f63f006b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63ef7fe640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f63f00d1d50 0x7f63f006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.796+0000 7f63eeffd640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f63f015d2e0 0x7f63f00d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:37886/0 (socket says 172.21.15.46:37886) 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63eeffd640 1 -- 172.21.15.46:0/3747708847 learned_addr learned my addr 172.21.15.46:0/3747708847 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63ee7fc640 1 -- 172.21.15.46:0/3747708847 >> v2:172.21.15.121:3300/0 conn(0x7f63f00d1d50 msgr2=0x7f63f006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63ee7fc640 1 --2- 172.21.15.46:0/3747708847 >> v2:172.21.15.121:3300/0 conn(0x7f63f00d1d50 0x7f63f006ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63ee7fc640 1 -- 172.21.15.46:0/3747708847 >> v1:172.21.15.121:6789/0 conn(0x7f63f015d2e0 legacy=0x7f63f00d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63ee7fc640 1 -- 172.21.15.46:0/3747708847 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f006c4d0 con 0x7f63f015d6c0 2023-12-16T21:16:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63edffb640 1 -- 172.21.15.46:0/3747708847 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111652929 0 0) 0x7f63f006ce00 con 0x7f63f015d2e0 2023-12-16T21:16:01.798 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.797+0000 7f63ef7fe640 1 --2- 172.21.15.46:0/3747708847 >> v2:172.21.15.121:3300/0 conn(0x7f63f00d1d50 0x7f63f006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:16:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63ee7fc640 1 --2- 172.21.15.46:0/3747708847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d6c0 0x7f63f006b4c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097a0 tx=0x7f63d802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846c6575bf42c7d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63edffb640 1 -- 172.21.15.46:0/3747708847 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8031780 con 0x7f63f015d6c0 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63edffb640 1 -- 172.21.15.46:0/3747708847 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d80318e0 con 0x7f63f015d6c0 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63edffb640 1 -- 172.21.15.46:0/3747708847 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d8031c70 con 0x7f63f015d6c0 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63f5889640 1 -- 172.21.15.46:0/3747708847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d6c0 msgr2=0x7f63f006b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63f5889640 1 --2- 172.21.15.46:0/3747708847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d6c0 0x7f63f006b4c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097a0 tx=0x7f63d802b820 comp rx=0 tx=0).stop 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63f5889640 1 -- 172.21.15.46:0/3747708847 shutdown_connections 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63f5889640 1 --2- 172.21.15.46:0/3747708847 >> v2:172.21.15.121:3300/0 conn(0x7f63f00d1d50 0x7f63f006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63f5889640 1 --2- 172.21.15.46:0/3747708847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d6c0 0x7f63f006b4c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.798+0000 7f63f5889640 1 -- 172.21.15.46:0/3747708847 >> 172.21.15.46:0/3747708847 conn(0x7f63f009c580 msgr2=0x7f63f00d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.799+0000 7f63f5889640 1 -- 172.21.15.46:0/3747708847 shutdown_connections 2023-12-16T21:16:01.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.799+0000 7f63f5889640 1 -- 172.21.15.46:0/3747708847 wait complete. 2023-12-16T21:16:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.800+0000 7f63f5889640 1 Processor -- start 2023-12-16T21:16:01.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.800+0000 7f63f5889640 1 -- start start 2023-12-16T21:16:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63f5889640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 0x7f63f00cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63f5889640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d2e0 0x7f63f01f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63f5889640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63f015d6c0 0x7f63f00cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.803 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63f5889640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f00b66d0 con 0x7f63f00d1d50 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63f5889640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f01f38e0 con 0x7f63f015d6c0 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63f5889640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f01f3bc0 con 0x7f63f015d2e0 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 0x7f63f00cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 0x7f63f00cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58562/0 (socket says 172.21.15.46:58562) 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 -- 172.21.15.46:0/3834909534 learned_addr learned my addr 172.21.15.46:0/3834909534 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63ee7fc640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d2e0 0x7f63f01f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63ef7fe640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63f015d6c0 0x7f63f00cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 -- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d2e0 msgr2=0x7f63f01f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d2e0 0x7f63f01f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 -- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63f015d6c0 msgr2=0x7f63f00cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:01.804 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63f015d6c0 0x7f63f00cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 -- 172.21.15.46:0/3834909534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0009a70 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63ef7fe640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63f015d6c0 0x7f63f00cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63eeffd640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 0x7f63f00cad40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c4d0 tx=0x7f63e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.801+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e80074e0 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.802+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e8007660 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.802+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8004a20 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.802+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d8009400 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.802+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f00b6970 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.803+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f0067420 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.803+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f63e8004b80 con 0x7f63f00d1d50 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.803+0000 7f63e7fff640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f63d006ae40 0x7f63d006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:01.805 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.804+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63e8088710 con 0x7f63f00d1d50 2023-12-16T21:16:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.804+0000 7f63ee7fc640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f63d006ae40 0x7f63d006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:01.806 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.804+0000 7f63ee7fc640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f63d006ae40 0x7f63d006d300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f63d8036000 tx=0x7f63d803e040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:01.807 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:01.808+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f63e8053870 con 0x7f63f00d1d50 2023-12-16T21:16:02.020 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.021+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f63f00d0950 con 0x7f63f00d1d50 2023-12-16T21:16:02.021 INFO:tasks.ceph.mon.a.smithi046.stderr:2023-12-16T21:16:02.021+0000 7ff1817f7640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:16:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.023+0000 7f63e7fff640 1 -- 172.21.15.46:0/3834909534 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7f63e8018090 con 0x7f63f00d1d50 2023-12-16T21:16:02.023 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f63d006ae40 msgr2=0x7f63d006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f63d006ae40 0x7f63d006d300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f63d8036000 tx=0x7f63d803e040 comp rx=0 tx=0).stop 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 msgr2=0x7f63f00cad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 0x7f63f00cad40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f63e800c4d0 tx=0x7f63e800c990 comp rx=0 tx=0).stop 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 shutdown_connections 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f63d006ae40 0x7f63d006d300 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63f015d6c0 0x7f63f00cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.026 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63f015d2e0 0x7f63f01f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 --2- 172.21.15.46:0/3834909534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63f00d1d50 0x7f63f00cad40 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 >> 172.21.15.46:0/3834909534 conn(0x7f63f009c580 msgr2=0x7f63f00ce010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:02.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 shutdown_connections 2023-12-16T21:16:02.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.026+0000 7f63f5889640 1 -- 172.21.15.46:0/3834909534 wait complete. 2023-12-16T21:16:02.027 INFO:tasks.ceph.mds.b.smithi121.stderr: -2578> 2023-12-16T21:15:37.610+0000 7f2dd5b09640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-16T21:16:02.037 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-16T21:16:02.056 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-12-16T21:16:02.146 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.146+0000 7f4bf2e6e640 1 Processor -- start 2023-12-16T21:16:02.147 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.146+0000 7f4bf2e6e640 1 -- start start 2023-12-16T21:16:02.149 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.147+0000 7f4bf2e6e640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4bec0ce640 0x7f4bec0b6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.147+0000 7f4bf2e6e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec0d2e80 con 0x7f4bec0d31c0 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.147+0000 7f4bf2e6e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec0d28a0 con 0x7f4bec0d19b0 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.147+0000 7f4bf2e6e640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec0cdc50 con 0x7f4bec0ce640 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.147+0000 7f4bf0c0a640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f4bec0d31c0 0x7f4bec0d35a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38684/0 (socket says 172.21.15.46:38684) 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.147+0000 7f4bf0c0a640 1 -- 172.21.15.46:0/4156379444 learned_addr learned my addr 172.21.15.46:0/4156379444 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4bf140b640 1 --2- 172.21.15.46:0/4156379444 >> v2:172.21.15.121:3300/0 conn(0x7f4bec0ce640 0x7f4bec0b6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096163936 0 0) 0x7f4bec0d2e80 con 0x7f4bec0d31c0 2023-12-16T21:16:02.150 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd8003520 con 0x7f4bec0d31c0 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660753255 0 0) 0x7f4bec0d28a0 con 0x7f4bec0d19b0 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bec0d2e80 con 0x7f4bec0d19b0 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4bf140b640 1 -- 172.21.15.46:0/4156379444 >> v1:172.21.15.121:6789/0 conn(0x7f4bec0d19b0 legacy=0x7f4bec0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4bf140b640 1 -- 172.21.15.46:0/4156379444 >> v1:172.21.15.46:6789/0 conn(0x7f4bec0d31c0 legacy=0x7f4bec0d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.148+0000 7f4bf140b640 1 -- 172.21.15.46:0/4156379444 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec0d2b10 con 0x7f4bec0ce640 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4bf140b640 1 --2- 172.21.15.46:0/4156379444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0ce640 0x7f4bec0b6470 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40097a0 tx=0x7f4bd402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a6aa81236d876c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4030610 con 0x7f4bec0ce640 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd4030770 con 0x7f4bec0ce640 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4beb7fe640 1 -- 172.21.15.46:0/4156379444 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4030b00 con 0x7f4bec0ce640 2023-12-16T21:16:02.151 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/4156379444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0ce640 msgr2=0x7f4bec0b6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/4156379444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0ce640 0x7f4bec0b6470 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40097a0 tx=0x7f4bd402b960 comp rx=0 tx=0).stop 2023-12-16T21:16:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/4156379444 shutdown_connections 2023-12-16T21:16:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.149+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/4156379444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0ce640 0x7f4bec0b6470 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.150+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/4156379444 >> 172.21.15.46:0/4156379444 conn(0x7f4bec072020 msgr2=0x7f4bec072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.150+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/4156379444 shutdown_connections 2023-12-16T21:16:02.152 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.150+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/4156379444 wait complete. 2023-12-16T21:16:02.155 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.151+0000 7f4bf2e6e640 1 Processor -- start 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.151+0000 7f4bf2e6e640 1 -- start start 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 0x7f4bec14e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bec0d19b0 0x7f4bec14ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0d31c0 0x7f4bec150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec14f770 con 0x7f4bec0ce640 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec14fa50 con 0x7f4bec0d31c0 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec14fd30 con 0x7f4bec0d19b0 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 0x7f4bec14e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 0x7f4bec14e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58564/0 (socket says 172.21.15.46:58564) 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 -- 172.21.15.46:0/1940445796 learned_addr learned my addr 172.21.15.46:0/1940445796 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:02.156 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bebfff640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bec0d19b0 0x7f4bec14ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf140b640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0d31c0 0x7f4bec150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 -- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bec0d19b0 msgr2=0x7f4bec14ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bec0d19b0 0x7f4bec14ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 -- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0d31c0 msgr2=0x7f4bec150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0d31c0 0x7f4bec150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 -- 172.21.15.46:0/1940445796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd4009400 con 0x7f4bec0ce640 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf140b640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0d31c0 0x7f4bec150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf0c0a640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 0x7f4bec14e360 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4be400cb20 tx=0x7f4be400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be400aca0 con 0x7f4bec0ce640 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4be400ae00 con 0x7f4bec0ce640 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec150010 con 0x7f4bec0ce640 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.152+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4be4012600 con 0x7f4bec0ce640 2023-12-16T21:16:02.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.153+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec15af40 con 0x7f4bec0ce640 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.153+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec150530 con 0x7f4bec0ce640 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.154+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f4be4007070 con 0x7f4bec0ce640 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.154+0000 7f4be97fa640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4bcc06ae70 0x7f4bcc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.154+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4be4089600 con 0x7f4bec0ce640 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.154+0000 7f4bebfff640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4bcc06ae70 0x7f4bcc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.155+0000 7f4bebfff640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4bcc06ae70 0x7f4bcc06d330 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0b2c70 tx=0x7f4bdc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:02.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.158+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f4bec150530 con 0x7f4bec0ce640 2023-12-16T21:16:02.372 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:02.372+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4bec150530 con 0x7f4bec0ce640 2023-12-16T21:16:03.024 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.024+0000 7f4be97fa640 1 -- 172.21.15.46:0/1940445796 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4bec150530 con 0x7f4bec0ce640 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4bcc06ae70 msgr2=0x7f4bcc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4bcc06ae70 0x7f4bcc06d330 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4bec0b2c70 tx=0x7f4bdc009040 comp rx=0 tx=0).stop 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 msgr2=0x7f4bec14e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 0x7f4bec14e360 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f4be400cb20 tx=0x7f4be400a060 comp rx=0 tx=0).stop 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 shutdown_connections 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f4bcc06ae70 0x7f4bcc06d330 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bec0d31c0 0x7f4bec150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bec0d19b0 0x7f4bec14ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 --2- 172.21.15.46:0/1940445796 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4bec0ce640 0x7f4bec14e360 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 >> 172.21.15.46:0/1940445796 conn(0x7f4bec072020 msgr2=0x7f4bec0cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 shutdown_connections 2023-12-16T21:16:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.027+0000 7f4bf2e6e640 1 -- 172.21.15.46:0/1940445796 wait complete. 2023-12-16T21:16:03.040 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-16T21:16:03.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.157+0000 7f6ef89e5640 1 Processor -- start 2023-12-16T21:16:03.157 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.157+0000 7f6ef89e5640 1 -- start start 2023-12-16T21:16:03.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef89e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef40b1400 0x7f6ef415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:03.158 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef89e5640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ef415e730 0x7f6ef415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef89e5640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef415e2a0 con 0x7f6ef4143490 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef89e5640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ef415b520 con 0x7f6ef415e730 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef89e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef4143870 con 0x7f6ef40b1400 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ef415e730 0x7f6ef415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef2d9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f6ef415e730 0x7f6ef415aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58572/0 (socket says 172.21.15.46:58572) 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef2d9d640 1 -- 172.21.15.46:0/1332521776 learned_addr learned my addr 172.21.15.46:0/1332521776 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef259c640 1 --1- 172.21.15.46:0/1332521776 >> v1:172.21.15.46:6789/0 conn(0x7f6ef4143490 0x7f6ef40b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38686/0 (socket says 172.21.15.46:38686) 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef1d9b640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef40b1400 0x7f6ef415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef2d9d640 1 -- 172.21.15.46:0/1332521776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef40b1400 msgr2=0x7f6ef415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef2d9d640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef40b1400 0x7f6ef415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef2d9d640 1 -- 172.21.15.46:0/1332521776 >> v1:172.21.15.46:6789/0 conn(0x7f6ef4143490 legacy=0x7f6ef40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.158+0000 7f6ef159a640 1 -- 172.21.15.46:0/1332521776 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144936778 0 0) 0x7f6ef415e2a0 con 0x7f6ef4143490 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.159+0000 7f6ef2d9d640 1 -- 172.21.15.46:0/1332521776 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef415b240 con 0x7f6ef415e730 2023-12-16T21:16:03.159 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.159+0000 7f6ef1d9b640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef40b1400 0x7f6ef415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.159+0000 7f6ef2d9d640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef415e730 0x7f6ef415aa20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009e50 tx=0x7f6ee402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe460bb35845b7a server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.159+0000 7f6ef159a640 1 -- 172.21.15.46:0/1332521776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee402cb00 con 0x7f6ef415e730 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.159+0000 7f6ef159a640 1 -- 172.21.15.46:0/1332521776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee402cc60 con 0x7f6ef415e730 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.159+0000 7f6ef159a640 1 -- 172.21.15.46:0/1332521776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee40343f0 con 0x7f6ef415e730 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 -- 172.21.15.46:0/1332521776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef415e730 msgr2=0x7f6ef415aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef415e730 0x7f6ef415aa20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4009e50 tx=0x7f6ee402bf60 comp rx=0 tx=0).stop 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 -- 172.21.15.46:0/1332521776 shutdown_connections 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef415e730 0x7f6ef415aa20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/1332521776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef40b1400 0x7f6ef415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 -- 172.21.15.46:0/1332521776 >> 172.21.15.46:0/1332521776 conn(0x7f6ef406a4d0 msgr2=0x7f6ef406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 -- 172.21.15.46:0/1332521776 shutdown_connections 2023-12-16T21:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.160+0000 7f6ef89e5640 1 -- 172.21.15.46:0/1332521776 wait complete. 2023-12-16T21:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.161+0000 7f6ef89e5640 1 Processor -- start 2023-12-16T21:16:03.162 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.161+0000 7f6ef89e5640 1 -- start start 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef89e5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef40b1400 0x7f6ef41399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef89e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 0x7f6ef4142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef89e5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef415e730 0x7f6ef413c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef89e5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef40ce450 con 0x7f6ef4143490 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef89e5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef4139ef0 con 0x7f6ef40b1400 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef89e5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef413f350 con 0x7f6ef415e730 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 0x7f6ef4142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 0x7f6ef4142830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58578/0 (socket says 172.21.15.46:58578) 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef1d9b640 1 -- 172.21.15.46:0/709366842 learned_addr learned my addr 172.21.15.46:0/709366842 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:03.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef2d9d640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef415e730 0x7f6ef413c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.162+0000 7f6ef259c640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef40b1400 0x7f6ef41399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef1d9b640 1 -- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef415e730 msgr2=0x7f6ef413c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef1d9b640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef415e730 0x7f6ef413c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef1d9b640 1 -- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef40b1400 msgr2=0x7f6ef41399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef1d9b640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef40b1400 0x7f6ef41399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef1d9b640 1 -- 172.21.15.46:0/709366842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4009b30 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef1d9b640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 0x7f6ef4142830 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00c7f0 tx=0x7f6edc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc007a30 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc007b90 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef413f630 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00ce40 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.163+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef40cde00 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.164+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ef4069400 con 0x7f6ef4143490 2023-12-16T21:16:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.164+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f6edc0113f0 con 0x7f6ef4143490 2023-12-16T21:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.165+0000 7f6eeaffd640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ed406ae40 0x7f6ed406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.165+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6edc0897d0 con 0x7f6ef4143490 2023-12-16T21:16:03.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.165+0000 7f6ef259c640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ed406ae40 0x7f6ed406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:03.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.166+0000 7f6ef259c640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ed406ae40 0x7f6ed406d300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6eec000f60 tx=0x7f6eec007b70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:03.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.169+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f6edc054820 con 0x7f6ef4143490 2023-12-16T21:16:03.345 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:03.345+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6ef40b0e70 con 0x7f6ef4143490 2023-12-16T21:16:04.025 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.026+0000 7f6eeaffd640 1 -- 172.21.15.46:0/709366842 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6edc01f0d0 con 0x7f6ef4143490 2023-12-16T21:16:04.026 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2023-12-16T21:16:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ed406ae40 msgr2=0x7f6ed406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ed406ae40 0x7f6ed406d300 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6eec000f60 tx=0x7f6eec007b70 comp rx=0 tx=0).stop 2023-12-16T21:16:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 msgr2=0x7f6ef4142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:04.029 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 0x7f6ef4142830 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00c7f0 tx=0x7f6edc00cbf0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 shutdown_connections 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f6ed406ae40 0x7f6ed406d300 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ef4143490 0x7f6ef4142830 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef415e730 0x7f6ef413c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 --2- 172.21.15.46:0/709366842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef40b1400 0x7f6ef41399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.029+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 >> 172.21.15.46:0/709366842 conn(0x7f6ef406a4d0 msgr2=0x7f6ef406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.030+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 shutdown_connections 2023-12-16T21:16:04.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.030+0000 7f6ef89e5640 1 -- 172.21.15.46:0/709366842 wait complete. 2023-12-16T21:16:04.044 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-16T21:16:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.161+0000 7fb718561640 1 Processor -- start 2023-12-16T21:16:04.161 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.161+0000 7fb718561640 1 -- start start 2023-12-16T21:16:04.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb718561640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 0x7fb7100b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb718561640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb71015abf0 0x7fb7100b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb718561640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71015e2a0 con 0x7fb71015e730 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb718561640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71015b520 con 0x7fb71015abf0 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb718561640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb710143870 con 0x7fb710143490 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb715afc640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb71015e730 0x7fb71015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38702/0 (socket says 172.21.15.46:38702) 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb715afc640 1 -- 172.21.15.46:0/1368429171 learned_addr learned my addr 172.21.15.46:0/1368429171 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb7162fd640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 0x7fb7100b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb716afe640 1 --2- 172.21.15.46:0/1368429171 >> v2:172.21.15.121:3300/0 conn(0x7fb71015abf0 0x7fb7100b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295679909 0 0) 0x7fb71015e2a0 con 0x7fb71015e730 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.162+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6fc003520 con 0x7fb71015e730 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2434821833 0 0) 0x7fb6fc003520 con 0x7fb71015e730 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 msgr2=0x7fb7100b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 0x7fb7100b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 >> v2:172.21.15.121:3300/0 conn(0x7fb71015abf0 msgr2=0x7fb7100b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 --2- 172.21.15.46:0/1368429171 >> v2:172.21.15.121:3300/0 conn(0x7fb71015abf0 0x7fb7100b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71015b240 con 0x7fb71015e730 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fb700001670 con 0x7fb71015e730 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 >> v1:172.21.15.46:6789/0 conn(0x7fb71015e730 legacy=0x7fb71015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6fc003970 0x7fb6fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 0x7fb6fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc004d00 0x7fb6fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71015b520 con 0x7fb6fc004d00 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb710143870 con 0x7fb6fc003970 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc00b980 con 0x7fb710143490 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb716afe640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc004d00 0x7fb6fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb7162fd640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6fc003970 0x7fb6fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb715afc640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 0x7fb6fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb716afe640 1 -- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 msgr2=0x7fb6fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb716afe640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb710143490 0x7fb6fc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb716afe640 1 -- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6fc003970 msgr2=0x7fb6fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb716afe640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6fc003970 0x7fb6fc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.163+0000 7fb716afe640 1 -- 172.21.15.46:0/1368429171 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc003520 con 0x7fb6fc004d00 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.164+0000 7fb716afe640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc004d00 0x7fb6fc005180 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb708009fc0 tx=0x7fb6fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65496613d6ac906 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.164+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70802cc60 con 0x7fb6fc004d00 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.164+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70800f040 con 0x7fb6fc004d00 2023-12-16T21:16:04.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.164+0000 7fb7152fb640 1 -- 172.21.15.46:0/1368429171 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb708034440 con 0x7fb6fc004d00 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb718561640 1 -- 172.21.15.46:0/1368429171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc004d00 msgr2=0x7fb6fc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb718561640 1 --2- 172.21.15.46:0/1368429171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6fc004d00 0x7fb6fc005180 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb708009fc0 tx=0x7fb6fc0057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb716afe640 1 -- 172.21.15.46:0/1368429171 reap_dead start 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb718561640 1 -- 172.21.15.46:0/1368429171 shutdown_connections 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb718561640 1 -- 172.21.15.46:0/1368429171 >> 172.21.15.46:0/1368429171 conn(0x7fb71006a4d0 msgr2=0x7fb71006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb718561640 1 -- 172.21.15.46:0/1368429171 shutdown_connections 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.165+0000 7fb718561640 1 -- 172.21.15.46:0/1368429171 wait complete. 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.166+0000 7fb718561640 1 Processor -- start 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.166+0000 7fb718561640 1 -- start start 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6fc004d00 0x7fb7100a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 0x7fb710070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7101595e0 0x7fb710159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.170 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71013e950 con 0x7fb7100caa90 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb710159fa0 con 0x7fb7101595e0 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7100a9420 con 0x7fb6fc004d00 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 0x7fb710070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 0x7fb710070600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58584/0 (socket says 172.21.15.46:58584) 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 -- 172.21.15.46:0/2788559603 learned_addr learned my addr 172.21.15.46:0/2788559603 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb716afe640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7101595e0 0x7fb710159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb7162fd640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6fc004d00 0x7fb7100a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 -- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6fc004d00 msgr2=0x7fb7100a8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6fc004d00 0x7fb7100a8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 -- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7101595e0 msgr2=0x7fb710159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7101595e0 0x7fb710159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 -- 172.21.15.46:0/2788559603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7100a9650 con 0x7fb7100caa90 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb716afe640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7101595e0 0x7fb710159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:04.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb715afc640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 0x7fb710070600 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb700007310 tx=0x7fb7000077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70000f9a0 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70000fb00 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7100a9930 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.167+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70000feb0 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.168+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7100a9ef0 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.168+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7100b4380 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.169+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb70001d460 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.169+0000 7fb706ffd640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb6f406ae40 0x7fb6f406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.170+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fb7000897f0 con 0x7fb7100caa90 2023-12-16T21:16:04.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.170+0000 7fb7162fd640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb6f406ae40 0x7fb6f406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.170+0000 7fb7162fd640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb6f406ae40 0x7fb6f406d300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009a80 tx=0x7fb70c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:04.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.174+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb7000549f0 con 0x7fb7100caa90 2023-12-16T21:16:04.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:04.355+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb7100c3fd0 con 0x7fb7100caa90 2023-12-16T21:16:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.027+0000 7fb706ffd640 1 -- 172.21.15.46:0/2788559603 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb7100c3fd0 con 0x7fb7100caa90 2023-12-16T21:16:05.027 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2023-12-16T21:16:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.030+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb6f406ae40 msgr2=0x7fb6f406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.030 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.030+0000 7fb718561640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb6f406ae40 0x7fb6f406d300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb70c009a80 tx=0x7fb70c008040 comp rx=0 tx=0).stop 2023-12-16T21:16:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.030+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 msgr2=0x7fb710070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.030+0000 7fb718561640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 0x7fb710070600 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb700007310 tx=0x7fb7000077d0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.031 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 shutdown_connections 2023-12-16T21:16:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb6f406ae40 0x7fb6f406d300 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7100caa90 0x7fb710070600 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7101595e0 0x7fb710159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 --2- 172.21.15.46:0/2788559603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6fc004d00 0x7fb7100a8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.032 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 >> 172.21.15.46:0/2788559603 conn(0x7fb71006a4d0 msgr2=0x7fb7100ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 shutdown_connections 2023-12-16T21:16:05.033 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.031+0000 7fb718561640 1 -- 172.21.15.46:0/2788559603 wait complete. 2023-12-16T21:16:05.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-16T21:16:05.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.161+0000 7efcf3048640 1 Processor -- start 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 -- start start 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcec0d31c0 0x7efcec0d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efcec0d19b0 0x7efcec0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec0ce470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec0d2e80 con 0x7efcec0d31c0 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec0d28a0 con 0x7efcec0d19b0 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf3048640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0d3ae0 con 0x7efcec0d1e20 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.162+0000 7efcf0de4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcec0d31c0 0x7efcec0d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7efcec0d31c0 0x7efcec0d35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58600/0 (socket says 172.21.15.46:58600) 2023-12-16T21:16:05.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 -- 172.21.15.46:0/448263568 learned_addr learned my addr 172.21.15.46:0/448263568 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf15e5640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec0ce470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcebfff640 1 --2- 172.21.15.46:0/448263568 >> v2:172.21.15.121:3300/0 conn(0x7efcec0d19b0 0x7efcec0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 -- 172.21.15.46:0/448263568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 msgr2=0x7efcec0ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec0ce470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 -- 172.21.15.46:0/448263568 >> v2:172.21.15.121:3300/0 conn(0x7efcec0d19b0 msgr2=0x7efcec0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 --2- 172.21.15.46:0/448263568 >> v2:172.21.15.121:3300/0 conn(0x7efcec0d19b0 0x7efcec0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf0de4640 1 -- 172.21.15.46:0/448263568 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0cdc50 con 0x7efcec0d31c0 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.163+0000 7efcf15e5640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec0ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf0de4640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec0d35a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efcec06bf30 tx=0x7efce002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8263f84dd8f7c031 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efceb7fe640 1 -- 172.21.15.46:0/448263568 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0031870 con 0x7efcec0d31c0 2023-12-16T21:16:05.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efceb7fe640 1 -- 172.21.15.46:0/448263568 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce00319d0 con 0x7efcec0d31c0 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efceb7fe640 1 -- 172.21.15.46:0/448263568 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0031d60 con 0x7efcec0d31c0 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 -- 172.21.15.46:0/448263568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 msgr2=0x7efcec0d35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec0d35a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7efcec06bf30 tx=0x7efce002b910 comp rx=0 tx=0).stop 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 -- 172.21.15.46:0/448263568 shutdown_connections 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec0ce470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 --2- 172.21.15.46:0/448263568 >> v2:172.21.15.121:3300/0 conn(0x7efcec0d19b0 0x7efcec0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 --2- 172.21.15.46:0/448263568 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec0d35a0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.164+0000 7efcf3048640 1 -- 172.21.15.46:0/448263568 >> 172.21.15.46:0/448263568 conn(0x7efcec072020 msgr2=0x7efcec072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.165+0000 7efcf3048640 1 -- 172.21.15.46:0/448263568 shutdown_connections 2023-12-16T21:16:05.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.165+0000 7efcf3048640 1 -- 172.21.15.46:0/448263568 wait complete. 2023-12-16T21:16:05.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.166+0000 7efcf3048640 1 Processor -- start 2023-12-16T21:16:05.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.166+0000 7efcf3048640 1 -- start start 2023-12-16T21:16:05.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec0d19b0 0x7efcec14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec1e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec1e5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1f2d10 con 0x7efcec0d31c0 2023-12-16T21:16:05.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0b4fb0 con 0x7efcec0d19b0 2023-12-16T21:16:05.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec1e2f10 con 0x7efcec0d1e20 2023-12-16T21:16:05.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec1e5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec1e5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58602/0 (socket says 172.21.15.46:58602) 2023-12-16T21:16:05.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 -- 172.21.15.46:0/2610748588 learned_addr learned my addr 172.21.15.46:0/2610748588 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:05.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcebfff640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec1e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf0de4640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec0d19b0 0x7efcec14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 -- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 msgr2=0x7efcec1e5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec1e5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 -- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec0d19b0 msgr2=0x7efcec14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec0d19b0 0x7efcec14e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 -- 172.21.15.46:0/2610748588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce00094d0 con 0x7efcec0d31c0 2023-12-16T21:16:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf15e5640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec1e5c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7efce4009fa0 tx=0x7efce400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4003b90 con 0x7efcec0d31c0 2023-12-16T21:16:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce400f040 con 0x7efcec0d31c0 2023-12-16T21:16:05.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec1e31f0 con 0x7efcec0d31c0 2023-12-16T21:16:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4004030 con 0x7efcec0d31c0 2023-12-16T21:16:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.167+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec14ea30 con 0x7efcec0d31c0 2023-12-16T21:16:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.169+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec1e3710 con 0x7efcec0d31c0 2023-12-16T21:16:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.169+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7efce4004190 con 0x7efcec0d31c0 2023-12-16T21:16:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.169+0000 7efce97fa640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efcd006ae40 0x7efcd006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.169+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efce4089780 con 0x7efcec0d31c0 2023-12-16T21:16:05.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.169+0000 7efcf0de4640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efcd006ae40 0x7efcd006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.170+0000 7efcf0de4640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efcd006ae40 0x7efcd006d300 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efce0036000 tx=0x7efce003c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:05.177 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.174+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7efce4054a90 con 0x7efcec0d31c0 2023-12-16T21:16:05.353 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.353+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7efcec069400 con 0x7efcec0d31c0 2023-12-16T21:16:05.355 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.356+0000 7efce97fa640 1 -- 172.21.15.46:0/2610748588 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7efce40580b0 con 0x7efcec0d31c0 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efcd006ae40 msgr2=0x7efcd006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efcd006ae40 0x7efcd006d300 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7efce0036000 tx=0x7efce003c040 comp rx=0 tx=0).stop 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 msgr2=0x7efcec1e5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec1e5c80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7efce4009fa0 tx=0x7efce400d560 comp rx=0 tx=0).stop 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 shutdown_connections 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7efcd006ae40 0x7efcd006d300 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcec0d31c0 0x7efcec1e5c80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.358 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0d1e20 0x7efcec1e5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 --2- 172.21.15.46:0/2610748588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcec0d19b0 0x7efcec14e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 >> 172.21.15.46:0/2610748588 conn(0x7efcec072020 msgr2=0x7efcec0cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:05.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 shutdown_connections 2023-12-16T21:16:05.359 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.358+0000 7efcf3048640 1 -- 172.21.15.46:0/2610748588 wait complete. 2023-12-16T21:16:05.369 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2023-12-16T21:16:05.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.476+0000 7fe93c851640 1 Processor -- start 2023-12-16T21:16:05.476 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.476+0000 7fe93c851640 1 -- start start 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93c851640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe938143490 0x7fe9380b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93c851640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe93815e730 0x7fe9380b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93c851640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe93815a4e0 0x7fe93815a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93c851640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93815b520 con 0x7fe93815e730 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93c851640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe938143870 con 0x7fe93815a4e0 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93c851640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93815e2a0 con 0x7fe938143490 2023-12-16T21:16:05.477 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe93815e730 0x7fe9380b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe93815e730 0x7fe9380b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58614/0 (socket says 172.21.15.46:58614) 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 -- 172.21.15.46:0/2043044402 learned_addr learned my addr 172.21.15.46:0/2043044402 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe93659c640 1 --2- 172.21.15.46:0/2043044402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe938143490 0x7fe9380b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe936d9d640 1 --2- 172.21.15.46:0/2043044402 >> v2:172.21.15.121:3300/0 conn(0x7fe93815a4e0 0x7fe93815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 -- 172.21.15.46:0/2043044402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe938143490 msgr2=0x7fe9380b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 --2- 172.21.15.46:0/2043044402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe938143490 0x7fe9380b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 -- 172.21.15.46:0/2043044402 >> v2:172.21.15.121:3300/0 conn(0x7fe93815a4e0 msgr2=0x7fe93815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 --2- 172.21.15.46:0/2043044402 >> v2:172.21.15.121:3300/0 conn(0x7fe93815a4e0 0x7fe93815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.478 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.477+0000 7fe92ffff640 1 -- 172.21.15.46:0/2043044402 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe93815b240 con 0x7fe93815e730 2023-12-16T21:16:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.478+0000 7fe92ffff640 1 --2- 172.21.15.46:0/2043044402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815e730 0x7fe9380b1230 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe93815eb90 tx=0x7fe92002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86e4bc160a634bc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.478+0000 7fe935d9b640 1 -- 172.21.15.46:0/2043044402 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031870 con 0x7fe93815e730 2023-12-16T21:16:05.479 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.478+0000 7fe935d9b640 1 -- 172.21.15.46:0/2043044402 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9200319d0 con 0x7fe93815e730 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.478+0000 7fe935d9b640 1 -- 172.21.15.46:0/2043044402 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031d60 con 0x7fe93815e730 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 -- 172.21.15.46:0/2043044402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815e730 msgr2=0x7fe9380b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 --2- 172.21.15.46:0/2043044402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815e730 0x7fe9380b1230 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe93815eb90 tx=0x7fe92002b910 comp rx=0 tx=0).stop 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 -- 172.21.15.46:0/2043044402 shutdown_connections 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 --2- 172.21.15.46:0/2043044402 >> v2:172.21.15.121:3300/0 conn(0x7fe93815a4e0 0x7fe93815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 --2- 172.21.15.46:0/2043044402 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815e730 0x7fe9380b1230 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 --2- 172.21.15.46:0/2043044402 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe938143490 0x7fe9380b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 -- 172.21.15.46:0/2043044402 >> 172.21.15.46:0/2043044402 conn(0x7fe93806a4d0 msgr2=0x7fe93806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 -- 172.21.15.46:0/2043044402 shutdown_connections 2023-12-16T21:16:05.480 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.479+0000 7fe93c851640 1 -- 172.21.15.46:0/2043044402 wait complete. 2023-12-16T21:16:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.480+0000 7fe93c851640 1 Processor -- start 2023-12-16T21:16:05.482 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.480+0000 7fe93c851640 1 -- start start 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93c851640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe938143490 0x7fe9381f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93c851640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 0x7fe9380a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93c851640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe93815e730 0x7fe9381f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93c851640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9381f1420 con 0x7fe93815a4e0 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93c851640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93813deb0 con 0x7fe938143490 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93c851640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93813e190 con 0x7fe93815e730 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 0x7fe9380a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe92ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 0x7fe9380a84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58624/0 (socket says 172.21.15.46:58624) 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe92ffff640 1 -- 172.21.15.46:0/2358660866 learned_addr learned my addr 172.21.15.46:0/2358660866 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe93659c640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe938143490 0x7fe9381f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.481+0000 7fe936d9d640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe93815e730 0x7fe9381f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92ffff640 1 -- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe93815e730 msgr2=0x7fe9381f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92ffff640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe93815e730 0x7fe9381f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.483 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92ffff640 1 -- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe938143490 msgr2=0x7fe9381f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92ffff640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe938143490 0x7fe9381f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92ffff640 1 -- 172.21.15.46:0/2358660866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200094d0 con 0x7fe93815a4e0 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92ffff640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 0x7fe9380a84d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe920000c00 tx=0x7fe9200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920031840 con 0x7fe93815a4e0 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9200443f0 con 0x7fe93815a4e0 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93813e470 con 0x7fe93815a4e0 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200447a0 con 0x7fe93815a4e0 2023-12-16T21:16:05.484 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.482+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93813ea30 con 0x7fe93815a4e0 2023-12-16T21:16:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.483+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93813ec20 con 0x7fe93815a4e0 2023-12-16T21:16:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.484+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe92004d450 con 0x7fe93815a4e0 2023-12-16T21:16:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.484+0000 7fe92effd640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe91806ae40 0x7fe91806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.484+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe92002d040 con 0x7fe93815a4e0 2023-12-16T21:16:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.484+0000 7fe93659c640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe91806ae40 0x7fe91806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:05.485 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.485+0000 7fe93659c640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe91806ae40 0x7fe91806d300 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe92800aac0 tx=0x7fe928008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:05.488 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.488+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe920076d70 con 0x7fe93815a4e0 2023-12-16T21:16:05.741 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:05.742+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) v1 -- 0x7fe938069400 con 0x7fe93815a4e0 2023-12-16T21:16:06.034 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.034+0000 7fe92effd640 1 -- 172.21.15.46:0/2358660866 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v163) v1 ==== 134+0+0 (secure 0 0 0) 0x7fe9200350c0 con 0x7fe93815a4e0 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe91806ae40 msgr2=0x7fe91806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe91806ae40 0x7fe91806d300 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe92800aac0 tx=0x7fe928008040 comp rx=0 tx=0).stop 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 msgr2=0x7fe9380a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 0x7fe9380a84d0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe920000c00 tx=0x7fe9200046c0 comp rx=0 tx=0).stop 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 shutdown_connections 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe91806ae40 0x7fe91806d300 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe93815a4e0 0x7fe9380a84d0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:06.036 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe93815e730 0x7fe9381f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 --2- 172.21.15.46:0/2358660866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe938143490 0x7fe9381f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 >> 172.21.15.46:0/2358660866 conn(0x7fe93806a4d0 msgr2=0x7fe9380b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 shutdown_connections 2023-12-16T21:16:06.037 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.036+0000 7fe93c851640 1 -- 172.21.15.46:0/2358660866 wait complete. 2023-12-16T21:16:06.048 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner:====================================================================== 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 775, in run_python 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: p.wait() 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait 2023-12-16T21:16:06.049 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:Ran 2 tests in 119.290s 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:====================================================================== 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2023-12-16T21:16:06.050 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 775, in run_python 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner: p.wait() 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi046 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2023-12-16T21:16:06.051 INFO:tasks.cephfs_test_runner: 2023-12-16T21:16:06.051 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-16T21:16:06.121 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1b5283e0716040b6ab0585e33cbafc9c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-16T21:16:06.123 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-12-16T21:16:06.132 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-12-16T21:16:06.178 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-12-16T21:16:06.178 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:16:06.178 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:16:06.210 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:16:06.211 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2023-12-16T21:16:06.211 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:16:06.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057149,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20507 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249480k,nr_inodes=812370,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498968k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-16T21:16:06.267 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:16:06.268 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:16:06.286 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:16:06.286 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-12-16T21:16:06.286 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:16:06.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056338,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3541 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248828k,nr_inodes=812207,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497660k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-16T21:16:06.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.344 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2023-12-16T21:16:06.361 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-16T21:16:06.361 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-16T21:16:06.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-16T21:16:06.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:16:06.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.508 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2023-12-16T21:16:06.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.610 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2023-12-16T21:16:06.628 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-12-16T21:16:06.629 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-16T21:16:06.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.629 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-16T21:16:06.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.700 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:16:06.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:16:06.770 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2023-12-16T21:16:06.873 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-16T21:16:06.883 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-16T21:16:06.884 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:16:06.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.994+0000 7f8ccdee7640 1 Processor -- start 2023-12-16T21:16:06.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.994+0000 7f8ccdee7640 1 -- start start 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8ccdee7640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cc8143490 0x7f8cc80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8ccdee7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8cc815e730 0x7f8cc80b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8ccdee7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc815e2a0 con 0x7f8cc815a4e0 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8ccdee7640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc815b520 con 0x7f8cc8143490 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8ccdee7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc8143870 con 0x7f8cc815e730 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8cc77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cc8143490 0x7f8cc80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:06.996 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8cc77fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f8cc8143490 0x7f8cc80b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58630/0 (socket says 172.21.15.46:58630) 2023-12-16T21:16:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.995+0000 7f8cc77fe640 1 -- 172.21.15.46:0/1013524913 learned_addr learned my addr 172.21.15.46:0/1013524913 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.996+0000 7f8cc6ffd640 1 --2- 172.21.15.46:0/1013524913 >> v2:172.21.15.121:3300/0 conn(0x7f8cc815e730 0x7f8cc80b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.996+0000 7f8cc77fe640 1 -- 172.21.15.46:0/1013524913 >> v2:172.21.15.121:3300/0 conn(0x7f8cc815e730 msgr2=0x7f8cc80b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.996+0000 7f8cc77fe640 1 --2- 172.21.15.46:0/1013524913 >> v2:172.21.15.121:3300/0 conn(0x7f8cc815e730 0x7f8cc80b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.996+0000 7f8cc77fe640 1 -- 172.21.15.46:0/1013524913 >> v1:172.21.15.121:6789/0 conn(0x7f8cc815a4e0 legacy=0x7f8cc815a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:06.997 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.996+0000 7f8cc77fe640 1 -- 172.21.15.46:0/1013524913 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc815b240 con 0x7f8cc8143490 2023-12-16T21:16:06.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8cc77fe640 1 --2- 172.21.15.46:0/1013524913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80b0cf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80aeca0 tx=0x7f8cb802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9178335c927cbd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:06.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8cc67fc640 1 -- 172.21.15.46:0/1013524913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8030860 con 0x7f8cc8143490 2023-12-16T21:16:06.999 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8cc67fc640 1 -- 172.21.15.46:0/1013524913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb80309c0 con 0x7f8cc8143490 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8cc67fc640 1 -- 172.21.15.46:0/1013524913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8030d50 con 0x7f8cc8143490 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 -- 172.21.15.46:0/1013524913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 msgr2=0x7f8cc80b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/1013524913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80b0cf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f8cc80aeca0 tx=0x7f8cb802b9a0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 -- 172.21.15.46:0/1013524913 shutdown_connections 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/1013524913 >> v2:172.21.15.121:3300/0 conn(0x7f8cc815e730 0x7f8cc80b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/1013524913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80b0cf0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 -- 172.21.15.46:0/1013524913 >> 172.21.15.46:0/1013524913 conn(0x7f8cc806a4d0 msgr2=0x7f8cc806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 -- 172.21.15.46:0/1013524913 shutdown_connections 2023-12-16T21:16:07.000 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.997+0000 7f8ccdee7640 1 -- 172.21.15.46:0/1013524913 wait complete. 2023-12-16T21:16:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.999+0000 7f8ccdee7640 1 Processor -- start 2023-12-16T21:16:07.001 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.999+0000 7f8ccdee7640 1 -- start start 2023-12-16T21:16:07.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:06.999+0000 7f8ccdee7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.003 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc815a4e0 0x7f8cc81585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc815e730 0x7f8cc80b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc80ac8e0 con 0x7f8cc8143490 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc8158eb0 con 0x7f8cc815e730 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc8159190 con 0x7f8cc815a4e0 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58640/0 (socket says 172.21.15.46:58640) 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 -- 172.21.15.46:0/4045869499 learned_addr learned my addr 172.21.15.46:0/4045869499 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:07.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc6ffd640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc815a4e0 0x7f8cc81585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 -- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc815a4e0 msgr2=0x7f8cc81585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc815a4e0 0x7f8cc81585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 -- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc815e730 msgr2=0x7f8cc80b0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc7fff640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc815e730 0x7f8cc80b0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc815e730 0x7f8cc80b0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 -- 172.21.15.46:0/4045869499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80094d0 con 0x7f8cc8143490 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8cc77fe640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80cb900 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8cb802b9e0 tx=0x7f8cb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8039c90 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8039e10 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8043640 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc8159470 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.000+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc8158b10 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.001+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc8069400 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.002+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8cb80437a0 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.002+0000 7f8caffff640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8cb406ae40 0x7f8cb406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.002+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8cb803e070 con 0x7f8cc8143490 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.002+0000 7f8cc6ffd640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8cb406ae40 0x7f8cb406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.003+0000 7f8cc6ffd640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8cb406ae40 0x7f8cb406d300 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8070a30 tx=0x7f8cbc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.007 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.006+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f8cb8077810 con 0x7f8cc8143490 2023-12-16T21:16:07.179 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.179+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8cc8158ca0 con 0x7f8cb406ae40 2023-12-16T21:16:07.181 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.182+0000 7f8caffff640 1 -- 172.21.15.46:0/4045869499 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50924 (secure 0 0 0) 0x7f8cc8158ca0 con 0x7f8cb406ae40 2023-12-16T21:16:07.182 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:16:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8cb406ae40 msgr2=0x7f8cb406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8cb406ae40 0x7f8cb406d300 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8070a30 tx=0x7f8cbc008040 comp rx=0 tx=0).stop 2023-12-16T21:16:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 msgr2=0x7f8cc80cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80cb900 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f8cb802b9e0 tx=0x7f8cb8004730 comp rx=0 tx=0).stop 2023-12-16T21:16:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 shutdown_connections 2023-12-16T21:16:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f8cb406ae40 0x7f8cb406d300 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc815e730 0x7f8cc80b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8cc815a4e0 0x7f8cc81585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 --2- 172.21.15.46:0/4045869499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cc8143490 0x7f8cc80cb900 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 >> 172.21.15.46:0/4045869499 conn(0x7f8cc806a4d0 msgr2=0x7f8cc815c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 shutdown_connections 2023-12-16T21:16:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.184+0000 7f8ccdee7640 1 -- 172.21.15.46:0/4045869499 wait complete. 2023-12-16T21:16:07.194 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2023-12-16T21:16:05.991210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238764,"kb_used_data":5960,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735956,"statfs":{"total":773094113280,"available":772849618944,"internally_reserved":0,"allocated":6103040,"data_stored":3223968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14789,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-64,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001329"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-16T21:15:36.635374+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:15:36.635374+0000","last_peered":"2023-12-16T21:15:36.635374+0000","last_clean":"2023-12-16T21:15:36.635374+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:15:36.635374+0000","last_undegraded":"2023-12-16T21:15:36.635374+0000","last_fullsized":"2023-12-16T21:15:36.635374+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":843143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.624}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67800000000000005,"5min":1.159,"15min":1.159},"last":0.53600000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69299999999999995,"5min":1.181,"15min":1.181},"last":0.51100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.495},{"interface":"front","average":{"1min":0.622,"5min":0.624,"15min":0.624},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.749,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56799999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67700000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.40300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48499999999999999,"15min":0.47199999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.755,"15min":0.755},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.625},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81399999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.42099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":252651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.64800000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.70899999999999996,"15min":0.73899999999999999},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46700000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.372},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46300000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.752,"15min":0.752},"last":0.48599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":253233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.434}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.60399999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.622},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.498}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":268968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.372}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":630784,"data_stored":268946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.309},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68700000000000006,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.34399999999999997}]},{"osd":1,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.433,"15min":0.438},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58499999999999996,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":1.367,"15min":1.367},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65700000000000003,"5min":1.46,"15min":1.46},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.874,"15min":0.874},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.62,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.621},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":602112,"data_stored":246730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.219}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.20599999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30299999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":843075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.377},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.318}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:16:07.196 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:16:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.303+0000 7f3a4b54c640 1 Processor -- start 2023-12-16T21:16:07.303 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.303+0000 7f3a4b54c640 1 -- start start 2023-12-16T21:16:07.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a4b54c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a4415e730 0x7f3a4415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a4b54c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4415e2a0 con 0x7f3a440b1400 2023-12-16T21:16:07.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a4b54c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4415b240 con 0x7f3a44143490 2023-12-16T21:16:07.305 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a4b54c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4415b520 con 0x7f3a4415e730 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a48ae7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f3a440b1400 0x7f3a4415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38718/0 (socket says 172.21.15.46:38718) 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a48ae7640 1 -- 172.21.15.46:0/1905500400 learned_addr learned my addr 172.21.15.46:0/1905500400 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a49ae9640 1 --2- 172.21.15.46:0/1905500400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a4415e730 0x7f3a4415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536707047 0 0) 0x7f3a4415e2a0 con 0x7f3a440b1400 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30003520 con 0x7f3a440b1400 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209105881 0 0) 0x7f3a4415b240 con 0x7f3a44143490 2023-12-16T21:16:07.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4415e2a0 con 0x7f3a44143490 2023-12-16T21:16:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147023916 0 0) 0x7f3a30003520 con 0x7f3a440b1400 2023-12-16T21:16:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a49ae9640 1 -- 172.21.15.46:0/1905500400 >> v1:172.21.15.121:6789/0 conn(0x7f3a44143490 legacy=0x7f3a440b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a49ae9640 1 -- 172.21.15.46:0/1905500400 >> v1:172.21.15.46:6789/0 conn(0x7f3a440b1400 legacy=0x7f3a4415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.304+0000 7f3a49ae9640 1 -- 172.21.15.46:0/1905500400 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4415dc50 con 0x7f3a4415e730 2023-12-16T21:16:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.305+0000 7f3a49ae9640 1 --2- 172.21.15.46:0/1905500400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a4415e730 0x7f3a4415abf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009dc0 tx=0x7f3a3c02bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26a786f9e67f9d37 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.305+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34001670 con 0x7f3a4415e730 2023-12-16T21:16:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.305+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c02c910 con 0x7f3a4415e730 2023-12-16T21:16:07.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.305+0000 7f3a3bfff640 1 -- 172.21.15.46:0/1905500400 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c02cc80 con 0x7f3a4415e730 2023-12-16T21:16:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 -- 172.21.15.46:0/1905500400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a4415e730 msgr2=0x7f3a4415abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/1905500400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a4415e730 0x7f3a4415abf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009dc0 tx=0x7f3a3c02bde0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 -- 172.21.15.46:0/1905500400 shutdown_connections 2023-12-16T21:16:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/1905500400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a4415e730 0x7f3a4415abf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 -- 172.21.15.46:0/1905500400 >> 172.21.15.46:0/1905500400 conn(0x7f3a4406a4d0 msgr2=0x7f3a4406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 -- 172.21.15.46:0/1905500400 shutdown_connections 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.306+0000 7f3a4b54c640 1 -- 172.21.15.46:0/1905500400 wait complete. 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.307+0000 7f3a4b54c640 1 Processor -- start 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.307+0000 7f3a4b54c640 1 -- start start 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a4b54c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a440b1400 0x7f3a44148900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a4b54c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 0x7f3a440aca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a4b54c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a4415e730 0x7f3a440af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a4b54c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a44168590 con 0x7f3a44143490 2023-12-16T21:16:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a4b54c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a44167910 con 0x7f3a4415e730 2023-12-16T21:16:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a4b54c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a44167bf0 con 0x7f3a440b1400 2023-12-16T21:16:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 0x7f3a440aca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 0x7f3a440aca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58646/0 (socket says 172.21.15.46:58646) 2023-12-16T21:16:07.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 -- 172.21.15.46:0/2948723015 learned_addr learned my addr 172.21.15.46:0/2948723015 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a49ae9640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a4415e730 0x7f3a440af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a492e8640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a440b1400 0x7f3a44148900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 -- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a440b1400 msgr2=0x7f3a44148900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a440b1400 0x7f3a44148900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 -- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a4415e730 msgr2=0x7f3a440af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a4415e730 0x7f3a440af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.308+0000 7f3a48ae7640 1 -- 172.21.15.46:0/2948723015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c009a70 con 0x7f3a44143490 2023-12-16T21:16:07.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.309+0000 7f3a48ae7640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 0x7f3a440aca70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400c750 tx=0x7f3a3400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.309+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34005ad0 con 0x7f3a44143490 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.309+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a34005c30 con 0x7f3a44143490 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.309+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a44167ed0 con 0x7f3a44143490 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.309+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34007070 con 0x7f3a44143490 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.309+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a440ae530 con 0x7f3a44143490 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.310+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a441686d0 con 0x7f3a44143490 2023-12-16T21:16:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.310+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f3a340071d0 con 0x7f3a44143490 2023-12-16T21:16:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.311+0000 7f3a39ffb640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3a2806ae40 0x7f3a2806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.311+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3a340888d0 con 0x7f3a44143490 2023-12-16T21:16:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.311+0000 7f3a492e8640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3a2806ae40 0x7f3a2806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.312+0000 7f3a492e8640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3a2806ae40 0x7f3a2806d300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a400011e0 tx=0x7f3a40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.315+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f3a441686d0 con 0x7f3a44143490 2023-12-16T21:16:07.487 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.487+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3a440b4f10 con 0x7f3a2806ae40 2023-12-16T21:16:07.490 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.490+0000 7f3a39ffb640 1 -- 172.21.15.46:0/2948723015 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50924 (secure 0 0 0) 0x7f3a440b4f10 con 0x7f3a2806ae40 2023-12-16T21:16:07.490 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:07.491 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:16:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3a2806ae40 msgr2=0x7f3a2806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.492 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3a2806ae40 0x7f3a2806d300 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a400011e0 tx=0x7f3a40009040 comp rx=0 tx=0).stop 2023-12-16T21:16:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 msgr2=0x7f3a440aca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 0x7f3a440aca70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400c750 tx=0x7f3a3400cc10 comp rx=0 tx=0).stop 2023-12-16T21:16:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 shutdown_connections 2023-12-16T21:16:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f3a2806ae40 0x7f3a2806d300 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a44143490 0x7f3a440aca70 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.493 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a4415e730 0x7f3a440af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 --2- 172.21.15.46:0/2948723015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a440b1400 0x7f3a44148900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 >> 172.21.15.46:0/2948723015 conn(0x7f3a4406a4d0 msgr2=0x7f3a44071320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 shutdown_connections 2023-12-16T21:16:07.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.492+0000 7f3a4b54c640 1 -- 172.21.15.46:0/2948723015 wait complete. 2023-12-16T21:16:07.502 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2023-12-16T21:16:05.991210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238764,"kb_used_data":5960,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735956,"statfs":{"total":773094113280,"available":772849618944,"internally_reserved":0,"allocated":6103040,"data_stored":3223968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14789,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-64,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001329"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-16T21:15:36.635374+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:15:36.635374+0000","last_peered":"2023-12-16T21:15:36.635374+0000","last_clean":"2023-12-16T21:15:36.635374+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:15:36.635374+0000","last_undegraded":"2023-12-16T21:15:36.635374+0000","last_fullsized":"2023-12-16T21:15:36.635374+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":843143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.624}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67800000000000005,"5min":1.159,"15min":1.159},"last":0.53600000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69299999999999995,"5min":1.181,"15min":1.181},"last":0.51100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.495},{"interface":"front","average":{"1min":0.622,"5min":0.624,"15min":0.624},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.749,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56799999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67700000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.40300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48499999999999999,"15min":0.47199999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.755,"15min":0.755},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.625},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81399999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.42099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":252651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.64800000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.70899999999999996,"15min":0.73899999999999999},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46700000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.372},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46300000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.752,"15min":0.752},"last":0.48599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":253233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.434}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.60399999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.622},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.498}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":268968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.372}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":630784,"data_stored":268946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.309},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68700000000000006,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.34399999999999997}]},{"osd":1,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.433,"15min":0.438},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58499999999999996,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":1.367,"15min":1.367},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65700000000000003,"5min":1.46,"15min":1.46},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.874,"15min":0.874},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.62,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.621},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":602112,"data_stored":246730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.219}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.20599999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30299999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":843075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.377},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.318}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:16:07.504 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-16T21:16:07.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:16:07.611 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.611+0000 7f2f5df1d640 1 Processor -- start 2023-12-16T21:16:07.612 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.611+0000 7f2f5df1d640 1 -- start start 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.612+0000 7f2f5df1d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f5815b130 0x7f2f580d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.612+0000 7f2f5df1d640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b3f10 con 0x7f2f580cf320 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.612+0000 7f2f5df1d640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580d0a50 con 0x7f2f5815f760 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.612+0000 7f2f5df1d640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5815fd20 con 0x7f2f5815b130 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.612+0000 7f2f577fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2f5815b130 0x7f2f580d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.612+0000 7f2f57fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2f580cf320 0x7f2f580cd310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38724/0 (socket says 172.21.15.46:38724) 2023-12-16T21:16:07.613 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.613+0000 7f2f57fff640 1 -- 172.21.15.46:0/597930504 learned_addr learned my addr 172.21.15.46:0/597930504 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.613+0000 7f2f577fe640 1 -- 172.21.15.46:0/597930504 >> v1:172.21.15.121:6789/0 conn(0x7f2f5815f760 legacy=0x7f2f580cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.613+0000 7f2f577fe640 1 -- 172.21.15.46:0/597930504 >> v1:172.21.15.46:6789/0 conn(0x7f2f580cf320 legacy=0x7f2f580cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.613+0000 7f2f577fe640 1 -- 172.21.15.46:0/597930504 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580d0d30 con 0x7f2f5815b130 2023-12-16T21:16:07.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.613+0000 7f2f567fc640 1 -- 172.21.15.46:0/597930504 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754107972 0 0) 0x7f2f580b3f10 con 0x7f2f580cf320 2023-12-16T21:16:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f577fe640 1 --2- 172.21.15.46:0/597930504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815b130 0x7f2f580d01f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2f480097a0 tx=0x7f2f4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2c1424110de846 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f567fc640 1 -- 172.21.15.46:0/597930504 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48031830 con 0x7f2f5815b130 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f567fc640 1 -- 172.21.15.46:0/597930504 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f48031990 con 0x7f2f5815b130 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f567fc640 1 -- 172.21.15.46:0/597930504 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f48031d20 con 0x7f2f5815b130 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 -- 172.21.15.46:0/597930504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815b130 msgr2=0x7f2f580d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/597930504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815b130 0x7f2f580d01f0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2f480097a0 tx=0x7f2f4802b8d0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 -- 172.21.15.46:0/597930504 shutdown_connections 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/597930504 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815b130 0x7f2f580d01f0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 -- 172.21.15.46:0/597930504 >> 172.21.15.46:0/597930504 conn(0x7f2f580b2c10 msgr2=0x7f2f580b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 -- 172.21.15.46:0/597930504 shutdown_connections 2023-12-16T21:16:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.614+0000 7f2f5df1d640 1 -- 172.21.15.46:0/597930504 wait complete. 2023-12-16T21:16:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.615+0000 7f2f5df1d640 1 Processor -- start 2023-12-16T21:16:07.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 -- start start 2023-12-16T21:16:07.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f580cf320 0x7f2f581f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f5815b130 0x7f2f581f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 0x7f2f5815d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815cce0 con 0x7f2f5815f760 2023-12-16T21:16:07.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815d2a0 con 0x7f2f5815b130 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f5df1d640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f581f3a20 con 0x7f2f580cf320 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.616+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 0x7f2f5815d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 0x7f2f5815d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58654/0 (socket says 172.21.15.46:58654) 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 -- 172.21.15.46:0/2838553934 learned_addr learned my addr 172.21.15.46:0/2838553934 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f577fe640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f580cf320 0x7f2f581f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f56ffd640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f5815b130 0x7f2f581f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 -- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f580cf320 msgr2=0x7f2f581f24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f580cf320 0x7f2f581f24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 -- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f5815b130 msgr2=0x7f2f581f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f5815b130 0x7f2f581f34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 -- 172.21.15.46:0/2838553934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f48009400 con 0x7f2f5815f760 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f577fe640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f580cf320 0x7f2f581f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f56ffd640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f5815b130 0x7f2f581f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:07.622 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f57fff640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 0x7f2f5815d730 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000c810 tx=0x7f2f5000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5000a5e0 con 0x7f2f5815f760 2023-12-16T21:16:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5000a760 con 0x7f2f5815f760 2023-12-16T21:16:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5000ab10 con 0x7f2f5815f760 2023-12-16T21:16:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f58159f30 con 0x7f2f5815f760 2023-12-16T21:16:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.617+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5815cf30 con 0x7f2f5815f760 2023-12-16T21:16:07.623 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.618+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f58067fc0 con 0x7f2f5815f760 2023-12-16T21:16:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.619+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2f5000ac70 con 0x7f2f5815f760 2023-12-16T21:16:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.619+0000 7f2f3ffff640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2f4406ae70 0x7f2f4406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.619+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2f50088770 con 0x7f2f5815f760 2023-12-16T21:16:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.619+0000 7f2f577fe640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2f4406ae70 0x7f2f4406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.620+0000 7f2f577fe640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2f4406ae70 0x7f2f4406d330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f48036000 tx=0x7f2f4803c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.623+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f2f50053b20 con 0x7f2f5815f760 2023-12-16T21:16:07.795 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.795+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f580cb970 con 0x7f2f4406ae70 2023-12-16T21:16:07.797 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.798+0000 7f2f3ffff640 1 -- 172.21.15.46:0/2838553934 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50924 (secure 0 0 0) 0x7f2f48031810 con 0x7f2f4406ae70 2023-12-16T21:16:07.798 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:07.798 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:16:07.799 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2f4406ae70 msgr2=0x7f2f4406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2f4406ae70 0x7f2f4406d330 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2f48036000 tx=0x7f2f4803c040 comp rx=0 tx=0).stop 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 msgr2=0x7f2f5815d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 0x7f2f5815d730 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000c810 tx=0x7f2f5000ccd0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 shutdown_connections 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2f4406ae70 0x7f2f4406d330 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f5815f760 0x7f2f5815d730 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f5815b130 0x7f2f581f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 --2- 172.21.15.46:0/2838553934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2f580cf320 0x7f2f581f24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.800 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 >> 172.21.15.46:0/2838553934 conn(0x7f2f580b2c10 msgr2=0x7f2f5815a700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 shutdown_connections 2023-12-16T21:16:07.801 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.800+0000 7f2f5df1d640 1 -- 172.21.15.46:0/2838553934 wait complete. 2023-12-16T21:16:07.810 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2023-12-16T21:16:05.991210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238764,"kb_used_data":5960,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735956,"statfs":{"total":773094113280,"available":772849618944,"internally_reserved":0,"allocated":6103040,"data_stored":3223968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14789,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-64,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001329"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-16T21:15:36.635374+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:15:36.635374+0000","last_peered":"2023-12-16T21:15:36.635374+0000","last_clean":"2023-12-16T21:15:36.635374+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:15:36.635374+0000","last_undegraded":"2023-12-16T21:15:36.635374+0000","last_fullsized":"2023-12-16T21:15:36.635374+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":843143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.624}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67800000000000005,"5min":1.159,"15min":1.159},"last":0.53600000000000003}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69299999999999995,"5min":1.181,"15min":1.181},"last":0.51100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.495},{"interface":"front","average":{"1min":0.622,"5min":0.624,"15min":0.624},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.749,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56799999999999995}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67700000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.38700000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.55300000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.40300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.44500000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48499999999999999,"15min":0.47199999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.755,"15min":0.755},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.41799999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.625},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.68799999999999994}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81399999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73399999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.42099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.78800000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":252651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.64800000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.70899999999999996,"15min":0.73899999999999999},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46700000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.60199999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.372},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.46300000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.52100000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.752,"15min":0.752},"last":0.48599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":253233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.372},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34100000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.434}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.64900000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.60399999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.622},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.498}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.67000000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":634880,"data_stored":268968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.35699999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.372}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.67600000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.623}]}]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31796,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340044,"statfs":{"total":96636764160,"available":96604205056,"internally_reserved":0,"allocated":630784,"data_stored":268946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.309},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68700000000000006,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.34399999999999997}]},{"osd":1,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.433,"15min":0.438},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58499999999999996,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":1.367,"15min":1.367},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65700000000000003,"5min":1.46,"15min":1.46},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.874,"15min":0.874},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.62,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.621},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":602112,"data_stored":246730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.219}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.20599999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30299999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.67900000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.56399999999999995}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64100000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1208320,"data_stored":843075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.377},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.318}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.36199999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.28999999999999998}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.46700000000000003}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.628}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:16:07.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:16:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.918+0000 7f8555b68640 1 Processor -- start 2023-12-16T21:16:07.918 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.918+0000 7f8555b68640 1 -- start start 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f8555b68640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f855015d330 0x7f85500b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f8555b68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 0x7f85500b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f8555b68640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85500b2b10 con 0x7f85500d2910 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f8555b68640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85500d1590 con 0x7f855015d330 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f8555b68640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85500d1870 con 0x7f85500d3900 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f854effd640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f85500d2910 0x7f85500d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38726/0 (socket says 172.21.15.46:38726) 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.919+0000 7f854effd640 1 -- 172.21.15.46:0/2861188253 learned_addr learned my addr 172.21.15.46:0/2861188253 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:07.922 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854ffff640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 0x7f85500b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854f7fe640 1 --2- 172.21.15.46:0/2861188253 >> v2:172.21.15.121:3300/0 conn(0x7f855015d330 0x7f85500b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137623981 0 0) 0x7f85500b2b10 con 0x7f85500d2910 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538003520 con 0x7f85500d2910 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1792050498 0 0) 0x7f8538003520 con 0x7f85500d2910 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 msgr2=0x7f85500b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 0x7f85500b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 >> v2:172.21.15.121:3300/0 conn(0x7f855015d330 msgr2=0x7f85500b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 --2- 172.21.15.46:0/2861188253 >> v2:172.21.15.121:3300/0 conn(0x7f855015d330 0x7f85500b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500d1b50 con 0x7f85500d2910 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f8544001670 con 0x7f85500d2910 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 >> v1:172.21.15.46:6789/0 conn(0x7f85500d2910 legacy=0x7f85500d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8538003970 0x7f8538003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380047c0 0x7f8538004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 0x7f8538005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500d1590 con 0x7f85380047c0 2023-12-16T21:16:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500d1870 con 0x7f8538003970 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853800b980 con 0x7f85500d3900 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854effd640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380047c0 0x7f8538004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854f7fe640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8538003970 0x7f8538003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854ffff640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 0x7f8538005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854effd640 1 -- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 msgr2=0x7f8538005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.920+0000 7f854effd640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85500d3900 0x7f8538005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854effd640 1 -- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8538003970 msgr2=0x7f8538003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854effd640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8538003970 0x7f8538003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854effd640 1 -- 172.21.15.46:0/2861188253 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538003520 con 0x7f85380047c0 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854effd640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380047c0 0x7f8538004c40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f854400c270 tx=0x7f85380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d15dfc899d6ea97 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854402cae0 con 0x7f85380047c0 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854402cc40 con 0x7f85380047c0 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.921+0000 7f854e7fc640 1 -- 172.21.15.46:0/2861188253 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85440058d0 con 0x7f85380047c0 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f8555b68640 1 -- 172.21.15.46:0/2861188253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380047c0 msgr2=0x7f8538004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f8555b68640 1 --2- 172.21.15.46:0/2861188253 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85380047c0 0x7f8538004c40 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f854400c270 tx=0x7f85380057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f854ffff640 1 -- 172.21.15.46:0/2861188253 reap_dead start 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f8555b68640 1 -- 172.21.15.46:0/2861188253 shutdown_connections 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f8555b68640 1 -- 172.21.15.46:0/2861188253 >> 172.21.15.46:0/2861188253 conn(0x7f8550072020 msgr2=0x7f8550072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f8555b68640 1 -- 172.21.15.46:0/2861188253 shutdown_connections 2023-12-16T21:16:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.922+0000 7f8555b68640 1 -- 172.21.15.46:0/2861188253 wait complete. 2023-12-16T21:16:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.923+0000 7f8555b68640 1 Processor -- start 2023-12-16T21:16:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.923+0000 7f8555b68640 1 -- start start 2023-12-16T21:16:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f8555b68640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85380047c0 0x7f85500cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f8555b68640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85501594f0 0x7f8550159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f8555b68640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 0x7f85501f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f8555b68640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500cd530 con 0x7f85501f1b40 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f8555b68640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500cc670 con 0x7f85380047c0 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f8555b68640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85500cc920 con 0x7f85501594f0 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 0x7f85501f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 0x7f85501f1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58674/0 (socket says 172.21.15.46:58674) 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 -- 172.21.15.46:0/1738033411 learned_addr learned my addr 172.21.15.46:0/1738033411 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 -- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85501594f0 msgr2=0x7f8550159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85501594f0 0x7f8550159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 -- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85380047c0 msgr2=0x7f85500cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85380047c0 0x7f85500cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.924+0000 7f854ffff640 1 -- 172.21.15.46:0/1738033411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500ccc00 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.925+0000 7f854ffff640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 0x7f85501f1fc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f855008c510 tx=0x7f854800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.925+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854800e040 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.925+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854800f040 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.925+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85500ccee0 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.925+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548015430 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.925+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8550159e70 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.926+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8550069400 con 0x7f85501f1b40 2023-12-16T21:16:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.926+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f8548015590 con 0x7f85501f1b40 2023-12-16T21:16:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.927+0000 7f8537fff640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f853c06ae70 0x7f853c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.927+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8548089c50 con 0x7f85501f1b40 2023-12-16T21:16:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.927+0000 7f854f7fe640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f853c06ae70 0x7f853c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.927+0000 7f854f7fe640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f853c06ae70 0x7f853c06d330 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8540006b10 tx=0x7f8540006aa0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:07.931+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f854808f020 con 0x7f85501f1b40 2023-12-16T21:16:08.109 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.109+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85501605e0 con 0x7f85501f1b40 2023-12-16T21:16:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.110+0000 7f8537fff640 1 -- 172.21.15.46:0/1738033411 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+12988 (secure 0 0 0) 0x7f8548055090 con 0x7f85501f1b40 2023-12-16T21:16:08.110 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:08.110 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":53,"fsid":"2397e6ee-7ffe-4438-b275-b4e10dea2d2f","created":"2023-12-16T21:13:11.738090+0000","modified":"2023-12-16T21:16:05.026539+0000","last_up_change":"2023-12-16T21:13:22.443049+0000","last_in_change":"2023-12-16T21:13:13.995107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:13:24.010064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8849da25-21f9-457e-ad40-67ca2672b6bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6801","nonce":2381230451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6803","nonce":2381230451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6807","nonce":2381230451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2381230451},{"type":"v1","addr":"172.21.15.46:6805","nonce":2381230451}]},"public_addr":"172.21.15.46:6801/2381230451","cluster_addr":"172.21.15.46:6803/2381230451","heartbeat_back_addr":"172.21.15.46:6807/2381230451","heartbeat_front_addr":"172.21.15.46:6805/2381230451","state":["exists","up"]},{"osd":1,"uuid":"2cbc7e60-c77c-40e8-b366-7213dbacbd92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6809","nonce":2214591015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6811","nonce":2214591015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6815","nonce":2214591015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2214591015},{"type":"v1","addr":"172.21.15.46:6813","nonce":2214591015}]},"public_addr":"172.21.15.46:6809/2214591015","cluster_addr":"172.21.15.46:6811/2214591015","heartbeat_back_addr":"172.21.15.46:6815/2214591015","heartbeat_front_addr":"172.21.15.46:6813/2214591015","state":["exists","up"]},{"osd":2,"uuid":"017ec04d-f0fd-4898-b169-0571b4ca3fe5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6817","nonce":2454235421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6819","nonce":2454235421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6823","nonce":2454235421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2454235421},{"type":"v1","addr":"172.21.15.46:6821","nonce":2454235421}]},"public_addr":"172.21.15.46:6817/2454235421","cluster_addr":"172.21.15.46:6819/2454235421","heartbeat_back_addr":"172.21.15.46:6823/2454235421","heartbeat_front_addr":"172.21.15.46:6821/2454235421","state":["exists","up"]},{"osd":3,"uuid":"5194ffa9-d0bb-4150-bfef-dafd3a848ec8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6825","nonce":1199415121}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6827","nonce":1199415121}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6831","nonce":1199415121}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1199415121},{"type":"v1","addr":"172.21.15.46:6829","nonce":1199415121}]},"public_addr":"172.21.15.46:6825/1199415121","cluster_addr":"172.21.15.46:6827/1199415121","heartbeat_back_addr":"172.21.15.46:6831/1199415121","heartbeat_front_addr":"172.21.15.46:6829/1199415121","state":["exists","up"]},{"osd":4,"uuid":"8fdcefd0-2beb-469a-a9b1-52f12ebc98cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6825","nonce":234804465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6827","nonce":234804465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6831","nonce":234804465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":234804465},{"type":"v1","addr":"172.21.15.121:6829","nonce":234804465}]},"public_addr":"172.21.15.121:6825/234804465","cluster_addr":"172.21.15.121:6827/234804465","heartbeat_back_addr":"172.21.15.121:6831/234804465","heartbeat_front_addr":"172.21.15.121:6829/234804465","state":["exists","up"]},{"osd":5,"uuid":"aa80e233-3e84-470b-8c10-bb8abe6f3058","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6809","nonce":1440061954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6811","nonce":1440061954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6818","nonce":1440061954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1440061954},{"type":"v1","addr":"172.21.15.121:6813","nonce":1440061954}]},"public_addr":"172.21.15.121:6809/1440061954","cluster_addr":"172.21.15.121:6811/1440061954","heartbeat_back_addr":"172.21.15.121:6818/1440061954","heartbeat_front_addr":"172.21.15.121:6813/1440061954","state":["exists","up"]},{"osd":6,"uuid":"2684c217-04a6-4b3e-9f3d-3e69c41432d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6815","nonce":3722378128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6819","nonce":3722378128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6823","nonce":3722378128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":3722378128},{"type":"v1","addr":"172.21.15.121:6821","nonce":3722378128}]},"public_addr":"172.21.15.121:6815/3722378128","cluster_addr":"172.21.15.121:6819/3722378128","heartbeat_back_addr":"172.21.15.121:6823/3722378128","heartbeat_front_addr":"172.21.15.121:6821/3722378128","state":["exists","up"]},{"osd":7,"uuid":"4c383297-1347-41c9-92a3-5cb522618343","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6801","nonce":2871933776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6803","nonce":2871933776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6807","nonce":2871933776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2871933776},{"type":"v1","addr":"172.21.15.121:6805","nonce":2871933776}]},"public_addr":"172.21.15.121:6801/2871933776","cluster_addr":"172.21.15.121:6803/2871933776","heartbeat_back_addr":"172.21.15.121:6807/2871933776","heartbeat_front_addr":"172.21.15.121:6805/2871933776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.054298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.163540+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.153742+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.215139+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.140478+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.141305+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.165197+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:13:19.200666+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/2015861199":"2023-12-17T21:16:02.022050+0000","172.21.15.121:6835/2015861199":"2023-12-17T21:16:02.022050+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f853c06ae70 msgr2=0x7f853c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f853c06ae70 0x7f853c06d330 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f8540006b10 tx=0x7f8540006aa0 comp rx=0 tx=0).stop 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 msgr2=0x7f85501f1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 0x7f85501f1fc0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f855008c510 tx=0x7f854800f580 comp rx=0 tx=0).stop 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 shutdown_connections 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f853c06ae70 0x7f853c06d330 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85501f1b40 0x7f85501f1fc0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85501594f0 0x7f8550159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 --2- 172.21.15.46:0/1738033411 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85380047c0 0x7f85500cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 >> 172.21.15.46:0/1738033411 conn(0x7f8550072020 msgr2=0x7f855015e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 shutdown_connections 2023-12-16T21:16:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:08.112+0000 7f8555b68640 1 -- 172.21.15.46:0/1738033411 wait complete. 2023-12-16T21:16:09.124 INFO:tasks.ceph:Scrubbing osd.0 2023-12-16T21:16:09.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:09.249 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.249+0000 7fe612389640 1 Processor -- start 2023-12-16T21:16:09.250 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.249+0000 7fe612389640 1 -- start start 2023-12-16T21:16:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe612389640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 0x7fe60c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe612389640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe60c0ceb70 0x7fe60c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe612389640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c071770 con 0x7fe60c0b4370 2023-12-16T21:16:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe612389640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60c0ccdc0 con 0x7fe60c0ceb70 2023-12-16T21:16:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe612389640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c0cd0a0 con 0x7fe60c144980 2023-12-16T21:16:09.251 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe610926640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe60c0b4370 0x7fe60c071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38732/0 (socket says 172.21.15.46:38732) 2023-12-16T21:16:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.250+0000 7fe610926640 1 -- 172.21.15.46:0/834072567 learned_addr learned my addr 172.21.15.46:0/834072567 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60bfff640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 0x7fe60c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60b7fe640 1 --2- 172.21.15.46:0/834072567 >> v2:172.21.15.121:3300/0 conn(0x7fe60c0ceb70 0x7fe60c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722573084 0 0) 0x7fe60c071770 con 0x7fe60c0b4370 2023-12-16T21:16:09.252 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f8003520 con 0x7fe60c0b4370 2023-12-16T21:16:09.258 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 396539184 0 0) 0x7fe5f8003520 con 0x7fe60c0b4370 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 msgr2=0x7fe60c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 0x7fe60c15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 >> v2:172.21.15.121:3300/0 conn(0x7fe60c0ceb70 msgr2=0x7fe60c0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 --2- 172.21.15.46:0/834072567 >> v2:172.21.15.121:3300/0 conn(0x7fe60c0ceb70 0x7fe60c0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c0cd380 con 0x7fe60c0b4370 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fe6040019f0 con 0x7fe60c0b4370 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 >> v1:172.21.15.46:6789/0 conn(0x7fe60c0b4370 legacy=0x7fe60c071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.259 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5f8003970 0x7fe5f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 0x7fe5f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f8004d00 0x7fe5f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c0ccdc0 con 0x7fe5f8004d00 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c0cd0a0 con 0x7fe5f8003970 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f800b980 con 0x7fe60c144980 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe610926640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f8004d00 0x7fe5f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60bfff640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5f8003970 0x7fe5f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.251+0000 7fe60b7fe640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 0x7fe5f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.260 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.252+0000 7fe610926640 1 -- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 msgr2=0x7fe5f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.252+0000 7fe610926640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c144980 0x7fe5f80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.252+0000 7fe610926640 1 -- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5f8003970 msgr2=0x7fe5f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.252+0000 7fe610926640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5f8003970 0x7fe5f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.252+0000 7fe610926640 1 -- 172.21.15.46:0/834072567 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8003520 con 0x7fe5f8004d00 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.252+0000 7fe610926640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f8004d00 0x7fe5f8005180 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe60400a8d0 tx=0x7fe5f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b2dfc2a2373a65 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604034630 con 0x7fe5f8004d00 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60402f070 con 0x7fe5f8004d00 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe60affd640 1 -- 172.21.15.46:0/834072567 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6040349d0 con 0x7fe5f8004d00 2023-12-16T21:16:09.261 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe612389640 1 -- 172.21.15.46:0/834072567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f8004d00 msgr2=0x7fe5f8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe612389640 1 --2- 172.21.15.46:0/834072567 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5f8004d00 0x7fe5f8005180 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe60400a8d0 tx=0x7fe5f80057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe610926640 1 -- 172.21.15.46:0/834072567 reap_dead start 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe612389640 1 -- 172.21.15.46:0/834072567 shutdown_connections 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe612389640 1 -- 172.21.15.46:0/834072567 >> 172.21.15.46:0/834072567 conn(0x7fe60c06a4d0 msgr2=0x7fe60c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe612389640 1 -- 172.21.15.46:0/834072567 shutdown_connections 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.253+0000 7fe612389640 1 -- 172.21.15.46:0/834072567 wait complete. 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.254+0000 7fe612389640 1 Processor -- start 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 -- start start 2023-12-16T21:16:09.262 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe60c1f6240 0x7fe60c0c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe60c0caca0 0x7fe60c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c1f6620 0x7fe60c141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c1559c0 con 0x7fe60c1f6240 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c0c9c20 con 0x7fe60c0caca0 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe612389640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c0c9f00 con 0x7fe60c1f6620 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe60bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe60c1f6240 0x7fe60c0c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe60bfff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe60c1f6240 0x7fe60c0c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58696/0 (socket says 172.21.15.46:58696) 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.255+0000 7fe60bfff640 1 -- 172.21.15.46:0/2455859999 learned_addr learned my addr 172.21.15.46:0/2455859999 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60b7fe640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe60c0caca0 0x7fe60c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.263 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe610926640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c1f6620 0x7fe60c141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60bfff640 1 -- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c1f6620 msgr2=0x7fe60c141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60bfff640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe60c1f6620 0x7fe60c141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60bfff640 1 -- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe60c0caca0 msgr2=0x7fe60c155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60bfff640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe60c0caca0 0x7fe60c155480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60bfff640 1 -- 172.21.15.46:0/2455859999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c0ca1e0 con 0x7fe60c1f6240 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe60bfff640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe60c1f6240 0x7fe60c0c9590 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40118e0 tx=0x7fe5f4011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f400fa30 con 0x7fe60c1f6240 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f400fb90 con 0x7fe60c1f6240 2023-12-16T21:16:09.264 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c0ca4c0 con 0x7fe60c1f6240 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4014440 con 0x7fe60c1f6240 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.256+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c142520 con 0x7fe60c1f6240 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.257+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe60c144d60 con 0x7fe60c1f6240 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe5f40145a0 con 0x7fe60c1f6240 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe608ff9640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe5ec06ae40 0x7fe5ec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe5f40899b0 con 0x7fe60c1f6240 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe608ff9640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fe5ec070ca0 0x7fe5ec0730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 --> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5ec073770 con 0x7fe5ec070ca0 2023-12-16T21:16:09.265 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5f408e2c0 con 0x7fe60c1f6240 2023-12-16T21:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe60b7fe640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe5ec06ae40 0x7fe5ec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.258+0000 7fe610926640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fe5ec070ca0 0x7fe5ec0730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.259+0000 7fe610926640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fe5ec070ca0 0x7fe5ec0730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.259+0000 7fe60b7fe640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe5ec06ae40 0x7fe5ec06d300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe60c156a90 tx=0x7fe5fc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.266 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.259+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== osd.0 v2:172.21.15.46:6800/2381230451 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe5ec073770 con 0x7fe5ec070ca0 2023-12-16T21:16:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 --> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe60c069400 con 0x7fe5ec070ca0 2023-12-16T21:16:09.273 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe608ff9640 1 -- 172.21.15.46:0/2455859999 <== osd.0 v2:172.21.15.46:6800/2381230451 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe60c069400 con 0x7fe5ec070ca0 2023-12-16T21:16:09.273 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:09.273 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:09.274 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:09.274 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fe5ec070ca0 msgr2=0x7fe5ec0730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:6800/2381230451,v1:172.21.15.46:6801/2381230451] conn(0x7fe5ec070ca0 0x7fe5ec0730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe5ec06ae40 msgr2=0x7fe5ec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe5ec06ae40 0x7fe5ec06d300 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe60c156a90 tx=0x7fe5fc009190 comp rx=0 tx=0).stop 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe60c1f6240 msgr2=0x7fe60c0c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 --2- 172.21.15.46:0/2455859999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe60c1f6240 0x7fe60c0c9590 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40118e0 tx=0x7fe5f4011ce0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe610926640 1 -- 172.21.15.46:0/2455859999 reap_dead start 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 shutdown_connections 2023-12-16T21:16:09.275 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 >> 172.21.15.46:0/2455859999 conn(0x7fe60c06a4d0 msgr2=0x7fe60c0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 shutdown_connections 2023-12-16T21:16:09.276 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.273+0000 7fe612389640 1 -- 172.21.15.46:0/2455859999 wait complete. 2023-12-16T21:16:09.287 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-16T21:16:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 Processor -- start 2023-12-16T21:16:09.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 -- start start 2023-12-16T21:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5810144980 0x7f581015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58100b4370 0x7f58100b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5810071770 con 0x7f58100ceb70 2023-12-16T21:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58100b4d20 con 0x7f58100b4370 2023-12-16T21:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.396+0000 7f5816889640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58100ccb60 con 0x7f5810144980 2023-12-16T21:16:09.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f5814e26640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f58100b4370 0x7f58100b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f580f7fe640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f58100ceb70 0x7f58100cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38746/0 (socket says 172.21.15.46:38746) 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f580f7fe640 1 -- 172.21.15.46:0/886373173 learned_addr learned my addr 172.21.15.46:0/886373173 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f580ffff640 1 --2- 172.21.15.46:0/886373173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5810144980 0x7f581015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f5814e26640 1 -- 172.21.15.46:0/886373173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5810144980 msgr2=0x7f581015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f5814e26640 1 --2- 172.21.15.46:0/886373173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5810144980 0x7f581015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f5814e26640 1 -- 172.21.15.46:0/886373173 >> v1:172.21.15.46:6789/0 conn(0x7f58100ceb70 legacy=0x7f58100cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f580effd640 1 -- 172.21.15.46:0/886373173 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843291851 0 0) 0x7f5810071770 con 0x7f58100ceb70 2023-12-16T21:16:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.397+0000 7f5814e26640 1 -- 172.21.15.46:0/886373173 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58100ccdc0 con 0x7f58100b4370 2023-12-16T21:16:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5814e26640 1 --2- 172.21.15.46:0/886373173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58100b4370 0x7f58100b47e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5808009d90 tx=0x7f580802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1745fbc90a12fca0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f580effd640 1 -- 172.21.15.46:0/886373173 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f580802cc50 con 0x7f58100b4370 2023-12-16T21:16:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f580effd640 1 -- 172.21.15.46:0/886373173 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f580802cdb0 con 0x7f58100b4370 2023-12-16T21:16:09.401 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f580effd640 1 -- 172.21.15.46:0/886373173 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808034660 con 0x7f58100b4370 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5816889640 1 -- 172.21.15.46:0/886373173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58100b4370 msgr2=0x7f58100b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5816889640 1 --2- 172.21.15.46:0/886373173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58100b4370 0x7f58100b47e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f5808009d90 tx=0x7f580802bea0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5816889640 1 -- 172.21.15.46:0/886373173 shutdown_connections 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5816889640 1 --2- 172.21.15.46:0/886373173 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f58100b4370 0x7f58100b47e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5816889640 1 --2- 172.21.15.46:0/886373173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5810144980 0x7f581015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.398+0000 7f5816889640 1 -- 172.21.15.46:0/886373173 >> 172.21.15.46:0/886373173 conn(0x7f581006a4d0 msgr2=0x7f581006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.399+0000 7f5816889640 1 -- 172.21.15.46:0/886373173 shutdown_connections 2023-12-16T21:16:09.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.399+0000 7f5816889640 1 -- 172.21.15.46:0/886373173 wait complete. 2023-12-16T21:16:09.405 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.400+0000 7f5816889640 1 Processor -- start 2023-12-16T21:16:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.400+0000 7f5816889640 1 -- start start 2023-12-16T21:16:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.400+0000 7f5816889640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58100b4370 0x7f5810155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.400+0000 7f5816889640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58100ceb70 0x7f581013ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.400+0000 7f5816889640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 0x7f5810158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.406 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5816889640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5810156350 con 0x7f5810144980 2023-12-16T21:16:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5816889640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5810156630 con 0x7f58100ceb70 2023-12-16T21:16:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5816889640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5810156910 con 0x7f58100b4370 2023-12-16T21:16:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 0x7f5810158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 0x7f5810158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58716/0 (socket says 172.21.15.46:58716) 2023-12-16T21:16:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 -- 172.21.15.46:0/984596102 learned_addr learned my addr 172.21.15.46:0/984596102 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f580f7fe640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58100ceb70 0x7f581013ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f580ffff640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58100b4370 0x7f5810155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 -- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58100b4370 msgr2=0x7f5810155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58100b4370 0x7f5810155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 -- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58100ceb70 msgr2=0x7f581013ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.408 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58100ceb70 0x7f581013ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 -- 172.21.15.46:0/984596102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5808009a70 con 0x7f5810144980 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5814e26640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 0x7f5810158020 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5808009d90 tx=0x7f5808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808040650 con 0x7f5810144980 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58080407d0 con 0x7f5810144980 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5810156bf0 con 0x7f5810144980 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5808040b80 con 0x7f5810144980 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.401+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581013bdc0 con 0x7f5810144980 2023-12-16T21:16:09.409 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.402+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58100cc9d0 con 0x7f5810144980 2023-12-16T21:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.403+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5808040ce0 con 0x7f5810144980 2023-12-16T21:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.403+0000 7f580cff9640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f57fc06ae40 0x7f57fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.410 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.403+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f58080ac220 con 0x7f5810144980 2023-12-16T21:16:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.403+0000 7f580ffff640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f57fc06ae40 0x7f57fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.404+0000 7f580ffff640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f57fc06ae40 0x7f57fc06d300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5800006fd0 tx=0x7f5800008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.411 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.407+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5808077540 con 0x7f5810144980 2023-12-16T21:16:09.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.582+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f58101567a0 con 0x7f57fc06ae40 2023-12-16T21:16:09.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.583+0000 7f580cff9640 1 -- 172.21.15.46:0/984596102 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f58101567a0 con 0x7f57fc06ae40 2023-12-16T21:16:09.583 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 0 to deep-scrub 2023-12-16T21:16:09.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f57fc06ae40 msgr2=0x7f57fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f57fc06ae40 0x7f57fc06d300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5800006fd0 tx=0x7f5800008040 comp rx=0 tx=0).stop 2023-12-16T21:16:09.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 msgr2=0x7f5810158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 0x7f5810158020 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5808009d90 tx=0x7f5808004a40 comp rx=0 tx=0).stop 2023-12-16T21:16:09.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 shutdown_connections 2023-12-16T21:16:09.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f57fc06ae40 0x7f57fc06d300 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5810144980 0x7f5810158020 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58100ceb70 0x7f581013ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 --2- 172.21.15.46:0/984596102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58100b4370 0x7f5810155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.584+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 >> 172.21.15.46:0/984596102 conn(0x7f581006a4d0 msgr2=0x7f581015ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.587 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.585+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 shutdown_connections 2023-12-16T21:16:09.588 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.585+0000 7f5816889640 1 -- 172.21.15.46:0/984596102 wait complete. 2023-12-16T21:16:09.597 INFO:tasks.ceph:Scrubbing osd.1 2023-12-16T21:16:09.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:09.707 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.707+0000 7f2a902cf640 1 Processor -- start 2023-12-16T21:16:09.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.707+0000 7f2a902cf640 1 -- start start 2023-12-16T21:16:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.708+0000 7f2a902cf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a880b1400 0x7f2a8815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.708+0000 7f2a902cf640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a8815e730 0x7f2a8815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.708+0000 7f2a902cf640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8815e2a0 con 0x7f2a88143490 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.708+0000 7f2a902cf640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8815b520 con 0x7f2a8815e730 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.708+0000 7f2a902cf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a88143870 con 0x7f2a880b1400 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e86c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f2a8815e730 0x7f2a8815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e06b640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f2a88143490 0x7f2a880b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38750/0 (socket says 172.21.15.46:38750) 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e06b640 1 -- 172.21.15.46:0/1965353335 learned_addr learned my addr 172.21.15.46:0/1965353335 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e86c640 1 -- 172.21.15.46:0/1965353335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a880b1400 msgr2=0x7f2a8815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8d86a640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a880b1400 0x7f2a8815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e86c640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a880b1400 0x7f2a8815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e86c640 1 -- 172.21.15.46:0/1965353335 >> v1:172.21.15.46:6789/0 conn(0x7f2a88143490 legacy=0x7f2a880b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8d069640 1 -- 172.21.15.46:0/1965353335 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2265553123 0 0) 0x7f2a8815e2a0 con 0x7f2a88143490 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8e86c640 1 -- 172.21.15.46:0/1965353335 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8815b240 con 0x7f2a8815e730 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.709+0000 7f2a8d86a640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a880b1400 0x7f2a8815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a8e86c640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a8815e730 0x7f2a8815aa20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009d90 tx=0x7f2a8002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2211b9fcedb769 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a8d069640 1 -- 172.21.15.46:0/1965353335 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8002cc50 con 0x7f2a8815e730 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a8d069640 1 -- 172.21.15.46:0/1965353335 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8002cdb0 con 0x7f2a8815e730 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a8d069640 1 -- 172.21.15.46:0/1965353335 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80034660 con 0x7f2a8815e730 2023-12-16T21:16:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 -- 172.21.15.46:0/1965353335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a8815e730 msgr2=0x7f2a8815aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a8815e730 0x7f2a8815aa20 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f2a80009d90 tx=0x7f2a8002bea0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 -- 172.21.15.46:0/1965353335 shutdown_connections 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a8815e730 0x7f2a8815aa20 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 --2- 172.21.15.46:0/1965353335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a880b1400 0x7f2a8815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 -- 172.21.15.46:0/1965353335 >> 172.21.15.46:0/1965353335 conn(0x7f2a8806a4d0 msgr2=0x7f2a8806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.710+0000 7f2a902cf640 1 -- 172.21.15.46:0/1965353335 shutdown_connections 2023-12-16T21:16:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.711+0000 7f2a902cf640 1 -- 172.21.15.46:0/1965353335 wait complete. 2023-12-16T21:16:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 Processor -- start 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 -- start start 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a880b1400 0x7f2a881effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a88143490 0x7f2a88140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a8815e730 0x7f2a881413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8814daa0 con 0x7f2a880b1400 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8814dd60 con 0x7f2a88143490 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.712+0000 7f2a902cf640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8814e040 con 0x7f2a8815e730 2023-12-16T21:16:09.717 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a880b1400 0x7f2a881effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a880b1400 0x7f2a881effe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58746/0 (socket says 172.21.15.46:58746) 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 -- 172.21.15.46:0/781050230 learned_addr learned my addr 172.21.15.46:0/781050230 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8d86a640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a88143490 0x7f2a88140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e86c640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a8815e730 0x7f2a881413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 -- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a8815e730 msgr2=0x7f2a881413f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a8815e730 0x7f2a881413f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 -- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a88143490 msgr2=0x7f2a88140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a88143490 0x7f2a88140eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.718 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 -- 172.21.15.46:0/781050230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a80009a70 con 0x7f2a880b1400 2023-12-16T21:16:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e86c640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a8815e730 0x7f2a881413f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8d86a640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a88143490 0x7f2a88140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a8e06b640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a880b1400 0x7f2a881effe0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400c4d0 tx=0x7f2a8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a840074e0 con 0x7f2a880b1400 2023-12-16T21:16:09.719 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a84007660 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8814e320 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a84004a20 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.713+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a88142680 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.714+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2a8815af60 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f2a84004b80 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a7effd640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a6c06ae40 0x7f2a6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2a84088850 con 0x7f2a880b1400 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a7effd640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f2a6c070ca0 0x7f2a6c0730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.720 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 --> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2a6c073770 con 0x7f2a6c070ca0 2023-12-16T21:16:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a8408d2c0 con 0x7f2a880b1400 2023-12-16T21:16:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a8d86a640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a6c06ae40 0x7f2a6c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.715+0000 7f2a8e86c640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f2a6c070ca0 0x7f2a6c0730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.716+0000 7f2a8e86c640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f2a6c070ca0 0x7f2a6c0730c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.716+0000 7f2a8d86a640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a6c06ae40 0x7f2a6c06d300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2a78006fd0 tx=0x7f2a78008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.721 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.716+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== osd.1 v2:172.21.15.46:6808/2214591015 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2a6c073770 con 0x7f2a6c070ca0 2023-12-16T21:16:09.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.729+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 --> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2a8815df30 con 0x7f2a6c070ca0 2023-12-16T21:16:09.730 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a7effd640 1 -- 172.21.15.46:0/781050230 <== osd.1 v2:172.21.15.46:6808/2214591015 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2a8002cb70 con 0x7f2a6c070ca0 2023-12-16T21:16:09.730 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:09.731 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:09.731 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f2a6c070ca0 msgr2=0x7f2a6c0730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:6808/2214591015,v1:172.21.15.46:6809/2214591015] conn(0x7f2a6c070ca0 0x7f2a6c0730c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a6c06ae40 msgr2=0x7f2a6c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f2a6c06ae40 0x7f2a6c06d300 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2a78006fd0 tx=0x7f2a78008040 comp rx=0 tx=0).stop 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a880b1400 msgr2=0x7f2a881effe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 --2- 172.21.15.46:0/781050230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a880b1400 0x7f2a881effe0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f2a8400c4d0 tx=0x7f2a8400c990 comp rx=0 tx=0).stop 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a8e86c640 1 -- 172.21.15.46:0/781050230 reap_dead start 2023-12-16T21:16:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 shutdown_connections 2023-12-16T21:16:09.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 >> 172.21.15.46:0/781050230 conn(0x7f2a8806a4d0 msgr2=0x7f2a8806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 shutdown_connections 2023-12-16T21:16:09.733 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.730+0000 7f2a902cf640 1 -- 172.21.15.46:0/781050230 wait complete. 2023-12-16T21:16:09.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-16T21:16:09.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.850+0000 7f5ddc34c640 1 Processor -- start 2023-12-16T21:16:09.850 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.850+0000 7f5ddc34c640 1 -- start start 2023-12-16T21:16:09.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5ddc34c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd40b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5ddc34c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5dd4150940 0x7f5dd40ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5ddc34c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd40b49f0 con 0x7f5dd40b3390 2023-12-16T21:16:09.852 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5ddc34c640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd40b3de0 con 0x7f5dd4150940 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5ddc34c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd40b40c0 con 0x7f5dd4150560 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dd98e7640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f5dd40b3390 0x7f5dd415d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38752/0 (socket says 172.21.15.46:38752) 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dd98e7640 1 -- 172.21.15.46:0/3596450314 learned_addr learned my addr 172.21.15.46:0/3596450314 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dda8e9640 1 --2- 172.21.15.46:0/3596450314 >> v2:172.21.15.46:3300/0 conn(0x7f5dd4150940 0x7f5dd40ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dda0e8640 1 --2- 172.21.15.46:0/3596450314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd40b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dd90e6640 1 -- 172.21.15.46:0/3596450314 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761120614 0 0) 0x7f5dd40b49f0 con 0x7f5dd40b3390 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dd90e6640 1 -- 172.21.15.46:0/3596450314 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc0003520 con 0x7f5dd40b3390 2023-12-16T21:16:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3596450314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 msgr2=0x7f5dd40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dda8e9640 1 --2- 172.21.15.46:0/3596450314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd40b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3596450314 >> v1:172.21.15.46:6789/0 conn(0x7f5dd40b3390 legacy=0x7f5dd415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.851+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3596450314 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd40b43a0 con 0x7f5dd4150940 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.852+0000 7f5dda8e9640 1 --2- 172.21.15.46:0/3596450314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4150940 0x7f5dd40ce0e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009d90 tx=0x7f5dcc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c135872e3dcc578 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.852+0000 7f5dd90e6640 1 -- 172.21.15.46:0/3596450314 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc02cc50 con 0x7f5dd4150940 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.852+0000 7f5dd90e6640 1 -- 172.21.15.46:0/3596450314 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc02cdb0 con 0x7f5dd4150940 2023-12-16T21:16:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.852+0000 7f5dd90e6640 1 -- 172.21.15.46:0/3596450314 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc034640 con 0x7f5dd4150940 2023-12-16T21:16:09.857 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3596450314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4150940 msgr2=0x7f5dd40ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3596450314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4150940 0x7f5dd40ce0e0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009d90 tx=0x7f5dcc02bea0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3596450314 shutdown_connections 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3596450314 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4150940 0x7f5dd40ce0e0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3596450314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd40b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3596450314 >> 172.21.15.46:0/3596450314 conn(0x7f5dd406a4d0 msgr2=0x7f5dd406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3596450314 shutdown_connections 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.853+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3596450314 wait complete. 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.854+0000 7f5ddc34c640 1 Processor -- start 2023-12-16T21:16:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.854+0000 7f5ddc34c640 1 -- start start 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5ddc34c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5dd40b3390 0x7f5dd41410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5ddc34c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd41674b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5ddc34c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 0x7f5dd4155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5ddc34c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd4155ac0 con 0x7f5dd4147990 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5ddc34c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd4148b70 con 0x7f5dd40b3390 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5ddc34c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd41554b0 con 0x7f5dd4150560 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 0x7f5dd4155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 0x7f5dd4155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58772/0 (socket says 172.21.15.46:58772) 2023-12-16T21:16:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3677233324 learned_addr learned my addr 172.21.15.46:0/3677233324 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda0e8640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5dd40b3390 0x7f5dd41410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 msgr2=0x7f5dd41674b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dd98e7640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd41674b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd41674b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5dd40b3390 msgr2=0x7f5dd41410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5dd40b3390 0x7f5dd41410b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 -- 172.21.15.46:0/3677233324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc009a70 con 0x7f5dd4147990 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.855+0000 7f5dda8e9640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 0x7f5dd4155f50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009eb0 tx=0x7f5dcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.856+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc03e610 con 0x7f5dd4147990 2023-12-16T21:16:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.856+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc03e770 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.856+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd4141d00 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.856+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dcc03eb60 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.856+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd4148e10 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.857+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd4155310 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.857+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5dcc03b050 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.857+0000 7f5dcaffd640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5db806ae40 0x7f5db806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.858+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5dcc0ab3e0 con 0x7f5dd4147990 2023-12-16T21:16:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.858+0000 7f5dda0e8640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5db806ae40 0x7f5db806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.858+0000 7f5dda0e8640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5db806ae40 0x7f5db806d300 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40cbaa0 tx=0x7f5dd0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:09.862+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5dd4155310 con 0x7f5dd4147990 2023-12-16T21:16:10.038 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.038+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dd40b20c0 con 0x7f5db806ae40 2023-12-16T21:16:10.039 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.039+0000 7f5dcaffd640 1 -- 172.21.15.46:0/3677233324 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5dd40b20c0 con 0x7f5db806ae40 2023-12-16T21:16:10.039 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 1 to deep-scrub 2023-12-16T21:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5db806ae40 msgr2=0x7f5db806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5db806ae40 0x7f5db806d300 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40cbaa0 tx=0x7f5dd0008040 comp rx=0 tx=0).stop 2023-12-16T21:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 msgr2=0x7f5dd4155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 0x7f5dd4155f50 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc009eb0 tx=0x7f5dcc004a40 comp rx=0 tx=0).stop 2023-12-16T21:16:10.041 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 shutdown_connections 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5db806ae40 0x7f5db806d300 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5dd4147990 0x7f5dd4155f50 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5dd4150560 0x7f5dd41674b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 --2- 172.21.15.46:0/3677233324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5dd40b3390 0x7f5dd41410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 >> 172.21.15.46:0/3677233324 conn(0x7f5dd406a4d0 msgr2=0x7f5dd40cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 shutdown_connections 2023-12-16T21:16:10.042 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.041+0000 7f5ddc34c640 1 -- 172.21.15.46:0/3677233324 wait complete. 2023-12-16T21:16:10.052 INFO:tasks.ceph:Scrubbing osd.2 2023-12-16T21:16:10.053 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.163+0000 7fe90a489640 1 Processor -- start 2023-12-16T21:16:10.163 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.163+0000 7fe90a489640 1 -- start start 2023-12-16T21:16:10.164 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe90a489640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9040ceb70 0x7fe9040cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe90a489640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe904071770 con 0x7fe904144980 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe90a489640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9040ccdc0 con 0x7fe9040b4370 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe90a489640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90415ced0 con 0x7fe9040ceb70 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe9037fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9040ceb70 0x7fe9040cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe903fff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fe904144980 0x7fe90415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:38764/0 (socket says 172.21.15.46:38764) 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe9037fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe9040ceb70 0x7fe9040cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58788/0 (socket says 172.21.15.46:58788) 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe903fff640 1 -- 172.21.15.46:0/776056972 learned_addr learned my addr 172.21.15.46:0/776056972 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe9037fe640 1 -- 172.21.15.46:0/776056972 >> v1:172.21.15.121:6789/0 conn(0x7fe9040b4370 legacy=0x7fe904071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:10.165 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe9037fe640 1 -- 172.21.15.46:0/776056972 >> v1:172.21.15.46:6789/0 conn(0x7fe904144980 legacy=0x7fe90415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.164+0000 7fe902ffd640 1 -- 172.21.15.46:0/776056972 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87604895 0 0) 0x7fe904071770 con 0x7fe904144980 2023-12-16T21:16:10.166 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.165+0000 7fe9037fe640 1 -- 172.21.15.46:0/776056972 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9040cd0a0 con 0x7fe9040ceb70 2023-12-16T21:16:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.165+0000 7fe9037fe640 1 --2- 172.21.15.46:0/776056972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040ceb70 0x7fe9040cc450 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40097a0 tx=0x7fe8f402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c302891eaaf20d4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.165+0000 7fe902ffd640 1 -- 172.21.15.46:0/776056972 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4030610 con 0x7fe9040ceb70 2023-12-16T21:16:10.167 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.165+0000 7fe902ffd640 1 -- 172.21.15.46:0/776056972 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f4030770 con 0x7fe9040ceb70 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.165+0000 7fe902ffd640 1 -- 172.21.15.46:0/776056972 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4030b00 con 0x7fe9040ceb70 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 -- 172.21.15.46:0/776056972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040ceb70 msgr2=0x7fe9040cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 --2- 172.21.15.46:0/776056972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040ceb70 0x7fe9040cc450 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe8f40097a0 tx=0x7fe8f402b960 comp rx=0 tx=0).stop 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 -- 172.21.15.46:0/776056972 shutdown_connections 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 --2- 172.21.15.46:0/776056972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040ceb70 0x7fe9040cc450 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 -- 172.21.15.46:0/776056972 >> 172.21.15.46:0/776056972 conn(0x7fe90406a4d0 msgr2=0x7fe90406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 -- 172.21.15.46:0/776056972 shutdown_connections 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.166+0000 7fe90a489640 1 -- 172.21.15.46:0/776056972 wait complete. 2023-12-16T21:16:10.168 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.167+0000 7fe90a489640 1 Processor -- start 2023-12-16T21:16:10.169 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.167+0000 7fe90a489640 1 -- start start 2023-12-16T21:16:10.171 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe90a489640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040b4370 0x7fe90413a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe90a489640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9040ceb70 0x7fe9040cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe90a489640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe904144980 0x7fe9040cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe90a489640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9041579f0 con 0x7fe9040b4370 2023-12-16T21:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe90a489640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415e230 con 0x7fe9040ceb70 2023-12-16T21:16:10.172 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe90a489640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe90415e7b0 con 0x7fe904144980 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe903fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040b4370 0x7fe90413a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe903fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040b4370 0x7fe90413a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58794/0 (socket says 172.21.15.46:58794) 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe903fff640 1 -- 172.21.15.46:0/1699276691 learned_addr learned my addr 172.21.15.46:0/1699276691 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe903fff640 1 -- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe904144980 msgr2=0x7fe9040cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe903fff640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe904144980 0x7fe9040cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe9037fe640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9040ceb70 0x7fe9040cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.168+0000 7fe903fff640 1 -- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9040ceb70 msgr2=0x7fe9040cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe903fff640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9040ceb70 0x7fe9040cac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.173 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe903fff640 1 -- 172.21.15.46:0/1699276691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f4009400 con 0x7fe9040b4370 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe9037fe640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9040ceb70 0x7fe9040cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe903fff640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040b4370 0x7fe90413a670 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00c3f0 tx=0x7fe8ec00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0074e0 con 0x7fe9040b4370 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec007640 con 0x7fe9040b4370 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9040ca690 con 0x7fe9040b4370 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec004a00 con 0x7fe9040b4370 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.169+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9040ca970 con 0x7fe9040b4370 2023-12-16T21:16:10.174 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.170+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe904144d60 con 0x7fe9040b4370 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.170+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe8ec004b60 con 0x7fe9040b4370 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe900ff9640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe8e406ae40 0x7fe8e406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe8ec089090 con 0x7fe9040b4370 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe900ff9640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7fe8e4070ca0 0x7fe8e40730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 --> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8e4073770 con 0x7fe8e4070ca0 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8ec08b750 con 0x7fe9040b4370 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe9037fe640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe8e406ae40 0x7fe8e406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.175 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.171+0000 7fe908a26640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7fe8e4070ca0 0x7fe8e40730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.172+0000 7fe908a26640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7fe8e4070ca0 0x7fe8e40730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.172+0000 7fe9037fe640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe8e406ae40 0x7fe8e406d300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4036000 tx=0x7fe8f403b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.172+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== osd.2 v2:172.21.15.46:6816/2454235421 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe8e4073770 con 0x7fe8e4070ca0 2023-12-16T21:16:10.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.185+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 --> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe904071400 con 0x7fe8e4070ca0 2023-12-16T21:16:10.185 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe900ff9640 1 -- 172.21.15.46:0/1699276691 <== osd.2 v2:172.21.15.46:6816/2454235421 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe8fc001d80 con 0x7fe8e4070ca0 2023-12-16T21:16:10.186 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:10.186 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:10.186 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7fe8e4070ca0 msgr2=0x7fe8e40730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:6816/2454235421,v1:172.21.15.46:6817/2454235421] conn(0x7fe8e4070ca0 0x7fe8e40730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe8e406ae40 msgr2=0x7fe8e406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe8e406ae40 0x7fe8e406d300 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4036000 tx=0x7fe8f403b040 comp rx=0 tx=0).stop 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040b4370 msgr2=0x7fe90413a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 --2- 172.21.15.46:0/1699276691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9040b4370 0x7fe90413a670 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00c3f0 tx=0x7fe8ec00c8b0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.187 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe908a26640 1 -- 172.21.15.46:0/1699276691 reap_dead start 2023-12-16T21:16:10.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 shutdown_connections 2023-12-16T21:16:10.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 >> 172.21.15.46:0/1699276691 conn(0x7fe90406a4d0 msgr2=0x7fe904157e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 shutdown_connections 2023-12-16T21:16:10.188 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.186+0000 7fe90a489640 1 -- 172.21.15.46:0/1699276691 wait complete. 2023-12-16T21:16:10.196 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.304+0000 7fb16b7d5640 1 Processor -- start 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.304+0000 7fb16b7d5640 1 -- start start 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb16b7d5640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb16415d3f0 0x7fb16406a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb16b7d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb16406aa10 0x7fb1640af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb16b7d5640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb164161ad0 con 0x7fb1640af560 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb16b7d5640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16406ae90 con 0x7fb16415d3f0 2023-12-16T21:16:10.306 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb16b7d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb164160ec0 con 0x7fb16406aa10 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb169571640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb16415d3f0 0x7fb16406a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb168d70640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb16406aa10 0x7fb1640af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb169d72640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb1640af560 0x7fb1640b3050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:37928/0 (socket says 172.21.15.46:37928) 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.305+0000 7fb169d72640 1 -- 172.21.15.46:0/2396806838 learned_addr learned my addr 172.21.15.46:0/2396806838 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb169571640 1 -- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb16406aa10 msgr2=0x7fb1640af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb169571640 1 --2- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb16406aa10 0x7fb1640af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb169571640 1 -- 172.21.15.46:0/2396806838 >> v1:172.21.15.121:6789/0 conn(0x7fb1640af560 legacy=0x7fb1640b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb169571640 1 -- 172.21.15.46:0/2396806838 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1641611a0 con 0x7fb16415d3f0 2023-12-16T21:16:10.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb168d70640 1 --2- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb16406aa10 0x7fb1640af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:16:10.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb15bfff640 1 -- 172.21.15.46:0/2396806838 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851404688 0 0) 0x7fb164161ad0 con 0x7fb1640af560 2023-12-16T21:16:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.306+0000 7fb169571640 1 --2- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16415d3f0 0x7fb16406a4d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb1600097a0 tx=0x7fb16002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb2663179dd16ea2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb15bfff640 1 -- 172.21.15.46:0/2396806838 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160031780 con 0x7fb16415d3f0 2023-12-16T21:16:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb15bfff640 1 -- 172.21.15.46:0/2396806838 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1600318e0 con 0x7fb16415d3f0 2023-12-16T21:16:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb15bfff640 1 -- 172.21.15.46:0/2396806838 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb160031c70 con 0x7fb16415d3f0 2023-12-16T21:16:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 -- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16415d3f0 msgr2=0x7fb16406a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16415d3f0 0x7fb16406a4d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb1600097a0 tx=0x7fb16002b820 comp rx=0 tx=0).stop 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 -- 172.21.15.46:0/2396806838 shutdown_connections 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb16406aa10 0x7fb1640af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/2396806838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16415d3f0 0x7fb16406a4d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 -- 172.21.15.46:0/2396806838 >> 172.21.15.46:0/2396806838 conn(0x7fb16406d3b0 msgr2=0x7fb16406d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 -- 172.21.15.46:0/2396806838 shutdown_connections 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.307+0000 7fb16b7d5640 1 -- 172.21.15.46:0/2396806838 wait complete. 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.308+0000 7fb16b7d5640 1 Processor -- start 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 -- start start 2023-12-16T21:16:10.313 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16406aa10 0x7fb1641511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1640af560 0x7fb164160340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 0x7fb1641501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415e2b0 con 0x7fb16415d3f0 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16415df80 con 0x7fb16406aa10 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb16b7d5640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1641517e0 con 0x7fb1640af560 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb169d72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 0x7fb1641501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb169d72640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 0x7fb1641501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58810/0 (socket says 172.21.15.46:58810) 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.309+0000 7fb169d72640 1 -- 172.21.15.46:0/149549858 learned_addr learned my addr 172.21.15.46:0/149549858 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:10.314 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb168d70640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1640af560 0x7fb164160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169571640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16406aa10 0x7fb1641511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169d72640 1 -- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1640af560 msgr2=0x7fb164160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169d72640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1640af560 0x7fb164160340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169d72640 1 -- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16406aa10 msgr2=0x7fb1641511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169d72640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16406aa10 0x7fb1641511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169d72640 1 -- 172.21.15.46:0/149549858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb154009400 con 0x7fb16415d3f0 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb169d72640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 0x7fb1641501e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00c7e0 tx=0x7fb15c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00a640 con 0x7fb16415d3f0 2023-12-16T21:16:10.315 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c00a7c0 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb160009400 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00ab70 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.310+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb164151da0 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.311+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16415e420 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.312+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb15c00acd0 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.312+0000 7fb159ffb640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb14806ae40 0x7fb14806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.312+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb15c088bb0 con 0x7fb16415d3f0 2023-12-16T21:16:10.316 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.313+0000 7fb169571640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb14806ae40 0x7fb14806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.313+0000 7fb169571640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb14806ae40 0x7fb14806d300 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb160036000 tx=0x7fb16003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.317 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.316+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb15c053f70 con 0x7fb16415d3f0 2023-12-16T21:16:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.491+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1640ad140 con 0x7fb14806ae40 2023-12-16T21:16:10.491 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.492+0000 7fb159ffb640 1 -- 172.21.15.46:0/149549858 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb1640ad140 con 0x7fb14806ae40 2023-12-16T21:16:10.492 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 2 to deep-scrub 2023-12-16T21:16:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb14806ae40 msgr2=0x7fb14806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb14806ae40 0x7fb14806d300 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb160036000 tx=0x7fb16003d040 comp rx=0 tx=0).stop 2023-12-16T21:16:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 msgr2=0x7fb1641501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 0x7fb1641501e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fb15c00c7e0 tx=0x7fb15c00cca0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 shutdown_connections 2023-12-16T21:16:10.494 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb14806ae40 0x7fb14806d300 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb16415d3f0 0x7fb1641501e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1640af560 0x7fb164160340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 --2- 172.21.15.46:0/149549858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb16406aa10 0x7fb1641511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 >> 172.21.15.46:0/149549858 conn(0x7fb16406d3b0 msgr2=0x7fb164157a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 shutdown_connections 2023-12-16T21:16:10.495 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.494+0000 7fb16b7d5640 1 -- 172.21.15.46:0/149549858 wait complete. 2023-12-16T21:16:10.504 INFO:tasks.ceph:Scrubbing osd.3 2023-12-16T21:16:10.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:10.614 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.615+0000 7f7420b9e640 1 Processor -- start 2023-12-16T21:16:10.615 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.615+0000 7f7420b9e640 1 -- start start 2023-12-16T21:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7420b9e640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f741c160fb0 0x7f741c161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7420b9e640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741c161d20 con 0x7f741c06ea50 2023-12-16T21:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7420b9e640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741c15e760 con 0x7f741c0d0ed0 2023-12-16T21:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7420b9e640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741c15d9a0 con 0x7f741c160fb0 2023-12-16T21:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7413fff640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f741c160fb0 0x7f741c161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.616 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f741a59c640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f741c06ea50 0x7f741c06ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50370/0 (socket says 172.21.15.46:50370) 2023-12-16T21:16:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f741a59c640 1 -- 172.21.15.46:0/2943042703 learned_addr learned my addr 172.21.15.46:0/2943042703 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7413fff640 1 -- 172.21.15.46:0/2943042703 >> v1:172.21.15.121:6789/0 conn(0x7f741c0d0ed0 legacy=0x7f741c0d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7413fff640 1 -- 172.21.15.46:0/2943042703 >> v1:172.21.15.46:6789/0 conn(0x7f741c06ea50 legacy=0x7f741c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7419d9b640 1 -- 172.21.15.46:0/2943042703 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139659360 0 0) 0x7f741c161d20 con 0x7f741c06ea50 2023-12-16T21:16:10.617 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.616+0000 7f7413fff640 1 -- 172.21.15.46:0/2943042703 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c15dc80 con 0x7f741c160fb0 2023-12-16T21:16:10.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.617+0000 7f7413fff640 1 --2- 172.21.15.46:0/2943042703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c161440 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f74040097a0 tx=0x7f740402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9493e21d1f769754 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.619 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.617+0000 7f7419d9b640 1 -- 172.21.15.46:0/2943042703 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740402e070 con 0x7f741c160fb0 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.617+0000 7f7419d9b640 1 -- 172.21.15.46:0/2943042703 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74040315a0 con 0x7f741c160fb0 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.617+0000 7f7419d9b640 1 -- 172.21.15.46:0/2943042703 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7404031930 con 0x7f741c160fb0 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.617+0000 7f7420b9e640 1 -- 172.21.15.46:0/2943042703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 msgr2=0x7f741c161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.617+0000 7f7420b9e640 1 --2- 172.21.15.46:0/2943042703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c161440 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f74040097a0 tx=0x7f740402b840 comp rx=0 tx=0).stop 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.618+0000 7f7420b9e640 1 -- 172.21.15.46:0/2943042703 shutdown_connections 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.618+0000 7f7420b9e640 1 --2- 172.21.15.46:0/2943042703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c161440 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.618+0000 7f7420b9e640 1 -- 172.21.15.46:0/2943042703 >> 172.21.15.46:0/2943042703 conn(0x7f741c06a6b0 msgr2=0x7f741c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.618+0000 7f7420b9e640 1 -- 172.21.15.46:0/2943042703 shutdown_connections 2023-12-16T21:16:10.620 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.618+0000 7f7420b9e640 1 -- 172.21.15.46:0/2943042703 wait complete. 2023-12-16T21:16:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.619+0000 7f7420b9e640 1 Processor -- start 2023-12-16T21:16:10.621 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.619+0000 7f7420b9e640 1 -- start start 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7420b9e640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c06ea50 0x7f741c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7420b9e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c0d0ed0 0x7f741c13f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7420b9e640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c0cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7420b9e640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c15b140 con 0x7f741c160fb0 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7420b9e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c0ca7a0 con 0x7f741c0d0ed0 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7420b9e640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741c0caa80 con 0x7f741c06ea50 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c0cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.624 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c0cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40898/0 (socket says 172.21.15.46:40898) 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 -- 172.21.15.46:0/1218139745 learned_addr learned my addr 172.21.15.46:0/1218139745 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741a59c640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c06ea50 0x7f741c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7413fff640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c0d0ed0 0x7f741c13f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 -- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c06ea50 msgr2=0x7f741c15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c06ea50 0x7f741c15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 -- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c0d0ed0 msgr2=0x7f741c13f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c0d0ed0 0x7f741c13f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 -- 172.21.15.46:0/1218139745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404009400 con 0x7f741c160fb0 2023-12-16T21:16:10.625 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741a59c640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c06ea50 0x7f741c15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7413fff640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f741c0d0ed0 0x7f741c13f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f741ad9d640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c0cd7d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f741400c780 tx=0x7f741400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.620+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414004b10 con 0x7f741c160fb0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.621+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414004c70 con 0x7f741c160fb0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.621+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741c0cad60 con 0x7f741c160fb0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.621+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741400aa90 con 0x7f741c160fb0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.621+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741c0cb320 con 0x7f741c160fb0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.621+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f741c15e240 con 0x7f741c160fb0 2023-12-16T21:16:10.626 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.622+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f741400abf0 con 0x7f741c160fb0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.622+0000 7f7412ffd640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73fc06ae40 0x7f73fc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.622+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7414089510 con 0x7f741c160fb0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f741a59c640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73fc06ae40 0x7f73fc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f7412ffd640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f73fc070ca0 0x7f73fc0730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 --> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73fc073770 con 0x7f73fc070ca0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f741408baf0 con 0x7f741c160fb0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f7413fff640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f73fc070ca0 0x7f73fc0730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f741a59c640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73fc06ae40 0x7f73fc06d300 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f740c000f60 tx=0x7f740c007ca0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.627 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.623+0000 7f7413fff640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f73fc070ca0 0x7f73fc0730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.628 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.624+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== osd.3 v2:172.21.15.46:6824/1199415121 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f73fc073770 con 0x7f73fc070ca0 2023-12-16T21:16:10.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.637+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 --> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f741c0cb510 con 0x7f73fc070ca0 2023-12-16T21:16:10.637 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.637+0000 7f7412ffd640 1 -- 172.21.15.46:0/1218139745 <== osd.3 v2:172.21.15.46:6824/1199415121 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f741c0cb510 con 0x7f73fc070ca0 2023-12-16T21:16:10.637 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:10.637 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:10.638 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.637+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f73fc070ca0 msgr2=0x7f73fc0730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.637+0000 7f7420b9e640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:6824/1199415121,v1:172.21.15.46:6825/1199415121] conn(0x7f73fc070ca0 0x7f73fc0730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73fc06ae40 msgr2=0x7f73fc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73fc06ae40 0x7f73fc06d300 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f740c000f60 tx=0x7f740c007ca0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 msgr2=0x7f741c0cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 --2- 172.21.15.46:0/1218139745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f741c160fb0 0x7f741c0cd7d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f741400c780 tx=0x7f741400cc40 comp rx=0 tx=0).stop 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f741ad9d640 1 -- 172.21.15.46:0/1218139745 reap_dead start 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 shutdown_connections 2023-12-16T21:16:10.639 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 >> 172.21.15.46:0/1218139745 conn(0x7f741c06a6b0 msgr2=0x7f741c15ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 shutdown_connections 2023-12-16T21:16:10.640 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.638+0000 7f7420b9e640 1 -- 172.21.15.46:0/1218139745 wait complete. 2023-12-16T21:16:10.648 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-16T21:16:10.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.755+0000 7f446ca56640 1 Processor -- start 2023-12-16T21:16:10.755 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.755+0000 7f446ca56640 1 -- start start 2023-12-16T21:16:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f446ca56640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4468160fb0 0x7f4468161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f446ca56640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4468161d20 con 0x7f44680d0ed0 2023-12-16T21:16:10.756 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f446ca56640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446815e760 con 0x7f446806ea50 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f446ca56640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446815d9a0 con 0x7f4468160fb0 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f4465d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4468160fb0 0x7f4468161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f4465d9b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f4468160fb0 0x7f4468161440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40900/0 (socket says 172.21.15.46:40900) 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f4466d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f44680d0ed0 0x7f44680d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50384/0 (socket says 172.21.15.46:50384) 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.756+0000 7f4465d9b640 1 -- 172.21.15.46:0/952179460 learned_addr learned my addr 172.21.15.46:0/952179460 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.757+0000 7f4465d9b640 1 -- 172.21.15.46:0/952179460 >> v1:172.21.15.121:6789/0 conn(0x7f446806ea50 legacy=0x7f446806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.757+0000 7f4465d9b640 1 -- 172.21.15.46:0/952179460 >> v1:172.21.15.46:6789/0 conn(0x7f44680d0ed0 legacy=0x7f44680d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.757 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.757+0000 7f4465d9b640 1 -- 172.21.15.46:0/952179460 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446815dc80 con 0x7f4468160fb0 2023-12-16T21:16:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.757+0000 7f4465d9b640 1 --2- 172.21.15.46:0/952179460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468160fb0 0x7f4468161440 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4450009870 tx=0x7f445002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c3c5c4d6ce68c8 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446559a640 1 -- 172.21.15.46:0/952179460 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450031870 con 0x7f4468160fb0 2023-12-16T21:16:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446559a640 1 -- 172.21.15.46:0/952179460 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44500319d0 con 0x7f4468160fb0 2023-12-16T21:16:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446559a640 1 -- 172.21.15.46:0/952179460 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450031d60 con 0x7f4468160fb0 2023-12-16T21:16:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 -- 172.21.15.46:0/952179460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468160fb0 msgr2=0x7f4468161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.759 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 --2- 172.21.15.46:0/952179460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468160fb0 0x7f4468161440 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f4450009870 tx=0x7f445002b910 comp rx=0 tx=0).stop 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 -- 172.21.15.46:0/952179460 shutdown_connections 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 --2- 172.21.15.46:0/952179460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4468160fb0 0x7f4468161440 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 -- 172.21.15.46:0/952179460 >> 172.21.15.46:0/952179460 conn(0x7f446806a6b0 msgr2=0x7f446806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 -- 172.21.15.46:0/952179460 shutdown_connections 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.758+0000 7f446ca56640 1 -- 172.21.15.46:0/952179460 wait complete. 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.759+0000 7f446ca56640 1 Processor -- start 2023-12-16T21:16:10.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 -- start start 2023-12-16T21:16:10.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 0x7f44681f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44680d0ed0 0x7f4468156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4468160fb0 0x7f446806ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446806cab0 con 0x7f446806ea50 2023-12-16T21:16:10.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44681f7d80 con 0x7f4468160fb0 2023-12-16T21:16:10.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446ca56640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446806bb90 con 0x7f44680d0ed0 2023-12-16T21:16:10.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 0x7f44681f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.765 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446659c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 0x7f44681f7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40912/0 (socket says 172.21.15.46:40912) 2023-12-16T21:16:10.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f446659c640 1 -- 172.21.15.46:0/321225138 learned_addr learned my addr 172.21.15.46:0/321225138 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:10.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f4465d9b640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44680d0ed0 0x7f4468156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.760+0000 7f4466d9d640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4468160fb0 0x7f446806ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.766 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446659c640 1 -- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44680d0ed0 msgr2=0x7f4468156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446659c640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44680d0ed0 0x7f4468156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446659c640 1 -- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4468160fb0 msgr2=0x7f446806ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446659c640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4468160fb0 0x7f446806ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446659c640 1 -- 172.21.15.46:0/321225138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500094d0 con 0x7f446806ea50 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446659c640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 0x7f44681f7840 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f446000b3f0 tx=0x7f446000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460011830 con 0x7f446806ea50 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4460011990 con 0x7f446806ea50 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446806be30 con 0x7f446806ea50 2023-12-16T21:16:10.767 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4460011d40 con 0x7f446806ea50 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.761+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446806c3f0 con 0x7f446806ea50 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.762+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468069400 con 0x7f446806ea50 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.763+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f446001a400 con 0x7f446806ea50 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.763+0000 7f445effd640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f444806ae40 0x7f444806d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.763+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4460088ff0 con 0x7f446806ea50 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.763+0000 7f4465d9b640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f444806ae40 0x7f444806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:10.768 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.764+0000 7f4465d9b640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f444806ae40 0x7f444806d300 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4450036000 tx=0x7f445003d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:10.769 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.767+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f44600544e0 con 0x7f446806ea50 2023-12-16T21:16:10.943 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.944+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4468157ca0 con 0x7f444806ae40 2023-12-16T21:16:10.944 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.944+0000 7f445effd640 1 -- 172.21.15.46:0/321225138 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4468157ca0 con 0x7f444806ae40 2023-12-16T21:16:10.944 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 3 to deep-scrub 2023-12-16T21:16:10.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f444806ae40 msgr2=0x7f444806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.946 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f444806ae40 0x7f444806d300 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4450036000 tx=0x7f445003d040 comp rx=0 tx=0).stop 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 msgr2=0x7f44681f7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 0x7f44681f7840 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f446000b3f0 tx=0x7f446000b8b0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 shutdown_connections 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f444806ae40 0x7f444806d300 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4468160fb0 0x7f446806ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44680d0ed0 0x7f4468156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 --2- 172.21.15.46:0/321225138 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f446806ea50 0x7f44681f7840 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 >> 172.21.15.46:0/321225138 conn(0x7f446806a6b0 msgr2=0x7f446815bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:10.947 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 shutdown_connections 2023-12-16T21:16:10.948 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:10.946+0000 7f446ca56640 1 -- 172.21.15.46:0/321225138 wait complete. 2023-12-16T21:16:10.957 INFO:tasks.ceph:Scrubbing osd.4 2023-12-16T21:16:10.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:11.068 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.069+0000 7f42c7623640 1 Processor -- start 2023-12-16T21:16:11.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.069+0000 7f42c7623640 1 -- start start 2023-12-16T21:16:11.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c7623640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f42c0160fb0 0x7f42c0161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c7623640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c0161d20 con 0x7f42c006ea50 2023-12-16T21:16:11.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c7623640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42c015e760 con 0x7f42c00d0ed0 2023-12-16T21:16:11.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c7623640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42c015d9a0 con 0x7f42c0160fb0 2023-12-16T21:16:11.072 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c53bf640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f42c006ea50 0x7f42c006ee30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50400/0 (socket says 172.21.15.46:50400) 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c53bf640 1 -- 172.21.15.46:0/3749400146 learned_addr learned my addr 172.21.15.46:0/3749400146 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/3749400146 >> v2:172.21.15.121:3300/0 conn(0x7f42c0160fb0 0x7f42c0161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913880990 0 0) 0x7f42c0161d20 con 0x7f42c006ea50 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42ac003520 con 0x7f42c006ea50 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176262161 0 0) 0x7f42ac003520 con 0x7f42c006ea50 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 >> v2:172.21.15.121:3300/0 conn(0x7f42c0160fb0 msgr2=0x7f42c0161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 --2- 172.21.15.46:0/3749400146 >> v2:172.21.15.121:3300/0 conn(0x7f42c0160fb0 0x7f42c0161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 >> v1:172.21.15.121:6789/0 conn(0x7f42c00d0ed0 legacy=0x7f42c00d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.073 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c015dc80 con 0x7f42c006ea50 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f42bc001670 con 0x7f42c006ea50 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 >> v1:172.21.15.46:6789/0 conn(0x7f42c006ea50 legacy=0x7f42c006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42ac003970 0x7f42ac003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac004370 0x7f42ac004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac0050e0 0x7f42ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c015d9a0 con 0x7f42ac004370 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac00c020 con 0x7f42ac003970 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.070+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac00c1e0 con 0x7f42ac0050e0 2023-12-16T21:16:11.074 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac004370 0x7f42ac004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c53bf640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42ac003970 0x7f42ac003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c5bc0640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac0050e0 0x7f42ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c4bbe640 1 -- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac0050e0 msgr2=0x7f42ac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac0050e0 0x7f42ac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c4bbe640 1 -- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42ac003970 msgr2=0x7f42ac003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42ac003970 0x7f42ac003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c4bbe640 1 -- 172.21.15.46:0/3749400146 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac003520 con 0x7f42ac004370 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.071+0000 7f42c5bc0640 1 -- 172.21.15.46:0/3749400146 reap_dead start 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac004370 0x7f42ac004840 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ac0 tx=0x7f42ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b1d8bad550a9d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.075 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002c6d0 con 0x7f42ac004370 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b002c830 con 0x7f42ac004370 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42b7fff640 1 -- 172.21.15.46:0/3749400146 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b002cbc0 con 0x7f42ac004370 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 -- 172.21.15.46:0/3749400146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac004370 msgr2=0x7f42ac004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac004370 0x7f42ac004840 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ac0 tx=0x7f42ac0057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 -- 172.21.15.46:0/3749400146 shutdown_connections 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 --2- 172.21.15.46:0/3749400146 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42ac004370 0x7f42ac004840 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 -- 172.21.15.46:0/3749400146 >> 172.21.15.46:0/3749400146 conn(0x7f42c006a6b0 msgr2=0x7f42c006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 -- 172.21.15.46:0/3749400146 shutdown_connections 2023-12-16T21:16:11.076 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.072+0000 7f42c7623640 1 -- 172.21.15.46:0/3749400146 wait complete. 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.073+0000 7f42c7623640 1 Processor -- start 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 -- start start 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac004370 0x7f42c01f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42c0151a40 0x7f42c006cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42c01540d0 0x7f42c01332f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0155630 con 0x7f42c0151a40 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0154a20 con 0x7f42c01540d0 2023-12-16T21:16:11.080 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c7623640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42c0154d00 con 0x7f42ac004370 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c4bbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42c0151a40 0x7f42c006cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c4bbe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42c0151a40 0x7f42c006cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40926/0 (socket says 172.21.15.46:40926) 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.074+0000 7f42c4bbe640 1 -- 172.21.15.46:0/192454600 learned_addr learned my addr 172.21.15.46:0/192454600 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c5bc0640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42c01540d0 0x7f42c01332f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c53bf640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac004370 0x7f42c01f30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c4bbe640 1 -- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac004370 msgr2=0x7f42c01f30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac004370 0x7f42c01f30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c4bbe640 1 -- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42c01540d0 msgr2=0x7f42c01332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.081 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42c01540d0 0x7f42c01332f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c4bbe640 1 -- 172.21.15.46:0/192454600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c0154fe0 con 0x7f42c0151a40 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c53bf640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f42ac004370 0x7f42c01f30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c5bc0640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f42c01540d0 0x7f42c01332f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c4bbe640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42c0151a40 0x7f42c006cda0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ac0 tx=0x7f42b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0003810 con 0x7f42c0151a40 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0003970 con 0x7f42c0151a40 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c01552c0 con 0x7f42c0151a40 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b0003d00 con 0x7f42c0151a40 2023-12-16T21:16:11.082 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.075+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c006d2e0 con 0x7f42c0151a40 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.076+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f42c015e240 con 0x7f42c0151a40 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f42b0040450 con 0x7f42c0151a40 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42b5ffb640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42a406ae40 0x7f42a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f42b00ab220 con 0x7f42c0151a40 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42b5ffb640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f42a4070ca0 0x7f42a40730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 --> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f42a4073770 con 0x7f42a4070ca0 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42b00ad8e0 con 0x7f42c0151a40 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.077+0000 7f42c53bf640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42a406ae40 0x7f42a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.083 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.078+0000 7f42c5bc0640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f42a4070ca0 0x7f42a40730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.078+0000 7f42c53bf640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42a406ae40 0x7f42a406d300 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0011e0 tx=0x7f42bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.078+0000 7f42c5bc0640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f42a4070ca0 0x7f42a40730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.084 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.079+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== osd.4 v2:172.21.15.121:6824/234804465 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f42a4073770 con 0x7f42a4070ca0 2023-12-16T21:16:11.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.092+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 --> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f42c0069400 con 0x7f42a4070ca0 2023-12-16T21:16:11.092 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.092+0000 7f42b5ffb640 1 -- 172.21.15.46:0/192454600 <== osd.4 v2:172.21.15.121:6824/234804465 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f42c0069400 con 0x7f42a4070ca0 2023-12-16T21:16:11.093 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:11.093 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:11.093 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f42a4070ca0 msgr2=0x7f42a40730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6824/234804465,v1:172.21.15.121:6825/234804465] conn(0x7f42a4070ca0 0x7f42a40730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.094 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42a406ae40 msgr2=0x7f42a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f42a406ae40 0x7f42a406d300 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0011e0 tx=0x7f42bc009040 comp rx=0 tx=0).stop 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42c0151a40 msgr2=0x7f42c006cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 --2- 172.21.15.46:0/192454600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42c0151a40 0x7f42c006cda0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009ac0 tx=0x7f42b0004730 comp rx=0 tx=0).stop 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c5bc0640 1 -- 172.21.15.46:0/192454600 reap_dead start 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 shutdown_connections 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 >> 172.21.15.46:0/192454600 conn(0x7f42c006a6b0 msgr2=0x7f42c013fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 shutdown_connections 2023-12-16T21:16:11.095 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.093+0000 7f42c7623640 1 -- 172.21.15.46:0/192454600 wait complete. 2023-12-16T21:16:11.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-16T21:16:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.211+0000 7fb1eeb89640 1 Processor -- start 2023-12-16T21:16:11.211 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.211+0000 7fb1eeb89640 1 -- start start 2023-12-16T21:16:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1eeb89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1e806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1eeb89640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80adae0 con 0x7fb1e815d1d0 2023-12-16T21:16:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1eeb89640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1e80addc0 con 0x7fb1e806bbf0 2023-12-16T21:16:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1eeb89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80ae0a0 con 0x7fb1e80ad640 2023-12-16T21:16:11.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1ec925640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7fb1e815d1d0 0x7fb1e806b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50410/0 (socket says 172.21.15.46:50410) 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1ec925640 1 -- 172.21.15.46:0/648451738 learned_addr learned my addr 172.21.15.46:0/648451738 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e7fff640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1e806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436137530 0 0) 0x7fb1e80adae0 con 0x7fb1e815d1d0 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d0003520 con 0x7fb1e815d1d0 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831715542 0 0) 0x7fb1d0003520 con 0x7fb1e815d1d0 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 msgr2=0x7fb1e806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1e806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 >> v1:172.21.15.121:6789/0 conn(0x7fb1e806bbf0 legacy=0x7fb1e806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80ae380 con 0x7fb1e815d1d0 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.212+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7fb1d8001670 con 0x7fb1e815d1d0 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 >> v1:172.21.15.46:6789/0 conn(0x7fb1e815d1d0 legacy=0x7fb1e806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1d00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1d0003ee0 0x7fb1d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d0004d00 0x7fb1d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e80ae0a0 con 0x7fb1d0004d00 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d000b9e0 con 0x7fb1d0003ee0 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d000bba0 con 0x7fb1e80ad640 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ec925640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1d00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1e7fff640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1d0003ee0 0x7fb1d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ed126640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d0004d00 0x7fb1d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ed126640 1 -- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 msgr2=0x7fb1d00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ed126640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1d00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ed126640 1 -- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1d0003ee0 msgr2=0x7fb1d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ed126640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1d0003ee0 0x7fb1d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ed126640 1 -- 172.21.15.46:0/648451738 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0003520 con 0x7fb1d0004d00 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.213+0000 7fb1ec925640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80ad640 0x7fb1d00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.214+0000 7fb1ed126640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d0004d00 0x7fb1d0005180 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80b4220 tx=0x7fb1d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68d962218c383dc server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.214+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e002d6b0 con 0x7fb1d0004d00 2023-12-16T21:16:11.219 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.214+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e002d830 con 0x7fb1d0004d00 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.214+0000 7fb1e77fe640 1 -- 172.21.15.46:0/648451738 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e002dbe0 con 0x7fb1d0004d00 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.214+0000 7fb1eeb89640 1 -- 172.21.15.46:0/648451738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d0004d00 msgr2=0x7fb1d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.214+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/648451738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1d0004d00 0x7fb1d0005180 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb1e80b4220 tx=0x7fb1d00057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.215+0000 7fb1ed126640 1 -- 172.21.15.46:0/648451738 reap_dead start 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.215+0000 7fb1eeb89640 1 -- 172.21.15.46:0/648451738 shutdown_connections 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.215+0000 7fb1eeb89640 1 -- 172.21.15.46:0/648451738 >> 172.21.15.46:0/648451738 conn(0x7fb1e8070e70 msgr2=0x7fb1e8071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.215+0000 7fb1eeb89640 1 -- 172.21.15.46:0/648451738 shutdown_connections 2023-12-16T21:16:11.220 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.215+0000 7fb1eeb89640 1 -- 172.21.15.46:0/648451738 wait complete. 2023-12-16T21:16:11.223 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.216+0000 7fb1eeb89640 1 Processor -- start 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.216+0000 7fb1eeb89640 1 -- start start 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1d0004d00 0x7fb1e8140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 0x7fb1e815f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80cf570 0x7fb1e806abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e81411c0 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e813e580 con 0x7fb1e80cf570 2023-12-16T21:16:11.224 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e813e860 con 0x7fb1d0004d00 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 0x7fb1e815f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 0x7fb1e815f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40938/0 (socket says 172.21.15.46:40938) 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 -- 172.21.15.46:0/4049697474 learned_addr learned my addr 172.21.15.46:0/4049697474 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1ec925640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1d0004d00 0x7fb1e8140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1ed126640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80cf570 0x7fb1e806abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 -- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1d0004d00 msgr2=0x7fb1e8140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.225 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1d0004d00 0x7fb1e8140c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 -- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80cf570 msgr2=0x7fb1e806abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80cf570 0x7fb1e806abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 -- 172.21.15.46:0/4049697474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e813eb40 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1ec925640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1d0004d00 0x7fb1e8140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e7fff640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 0x7fb1e815f660 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0119a0 tx=0x7fb1dc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00faf0 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1dc00fc50 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.226 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e813ee20 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc014440 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.217+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e814ea00 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.218+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e8069400 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.219+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb1dc0145a0 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.219+0000 7fb1e57fa640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1d406ae40 0x7fb1d406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.219+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb1dc089bf0 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.219+0000 7fb1ec925640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1d406ae40 0x7fb1d406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.227 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.220+0000 7fb1ec925640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1d406ae40 0x7fb1d406d300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8001030 tx=0x7fb1d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.228 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.223+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fb1dc054fa0 con 0x7fb1e80cf0f0 2023-12-16T21:16:11.399 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.399+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1e806dd00 con 0x7fb1d406ae40 2023-12-16T21:16:11.400 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.400+0000 7fb1e57fa640 1 -- 172.21.15.46:0/4049697474 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb1d8007770 con 0x7fb1d406ae40 2023-12-16T21:16:11.400 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 4 to deep-scrub 2023-12-16T21:16:11.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1d406ae40 msgr2=0x7fb1d406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1d406ae40 0x7fb1d406d300 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8001030 tx=0x7fb1d8009040 comp rx=0 tx=0).stop 2023-12-16T21:16:11.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 msgr2=0x7fb1e815f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.402 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 0x7fb1e815f660 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0119a0 tx=0x7fb1dc011e60 comp rx=0 tx=0).stop 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 shutdown_connections 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb1d406ae40 0x7fb1d406d300 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb1e80cf0f0 0x7fb1e815f660 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80cf570 0x7fb1e806abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 --2- 172.21.15.46:0/4049697474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1d0004d00 0x7fb1e8140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 >> 172.21.15.46:0/4049697474 conn(0x7fb1e8070e70 msgr2=0x7fb1e806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 shutdown_connections 2023-12-16T21:16:11.403 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.402+0000 7fb1eeb89640 1 -- 172.21.15.46:0/4049697474 wait complete. 2023-12-16T21:16:11.413 INFO:tasks.ceph:Scrubbing osd.5 2023-12-16T21:16:11.413 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:11.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.524+0000 7fb424d43640 1 Processor -- start 2023-12-16T21:16:11.524 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.524+0000 7fb424d43640 1 -- start start 2023-12-16T21:16:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb424d43640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42015d2d0 0x7fb4200b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb424d43640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb420160f70 0x7fb4200cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb424d43640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4201614e0 con 0x7fb4200ce480 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb424d43640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4201617c0 con 0x7fb420160f70 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb424d43640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb420161aa0 con 0x7fb42015d2d0 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41dd9b640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb420160f70 0x7fb4200cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41e59c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42015d2d0 0x7fb4200b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41ed9d640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb4200ce480 0x7fb42006b570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.46:33070/0 (socket says 172.21.15.46:33070) 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41ed9d640 1 -- 172.21.15.46:0/2726607506 learned_addr learned my addr 172.21.15.46:0/2726607506 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41e59c640 1 -- 172.21.15.46:0/2726607506 >> v2:172.21.15.121:3300/0 conn(0x7fb420160f70 msgr2=0x7fb4200cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41e59c640 1 --2- 172.21.15.46:0/2726607506 >> v2:172.21.15.121:3300/0 conn(0x7fb420160f70 0x7fb4200cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41e59c640 1 -- 172.21.15.46:0/2726607506 >> v1:172.21.15.121:6789/0 conn(0x7fb4200ce480 legacy=0x7fb42006b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.525+0000 7fb41e59c640 1 -- 172.21.15.46:0/2726607506 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420161d80 con 0x7fb42015d2d0 2023-12-16T21:16:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.526+0000 7fb41e59c640 1 --2- 172.21.15.46:0/2726607506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42015d2d0 0x7fb4200b8b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb4180097a0 tx=0x7fb41802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=456f9f2c30b9db3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.526+0000 7fb41d59a640 1 -- 172.21.15.46:0/2726607506 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418031830 con 0x7fb42015d2d0 2023-12-16T21:16:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.526+0000 7fb41d59a640 1 -- 172.21.15.46:0/2726607506 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb418031990 con 0x7fb42015d2d0 2023-12-16T21:16:11.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb41d59a640 1 -- 172.21.15.46:0/2726607506 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418031d20 con 0x7fb42015d2d0 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 -- 172.21.15.46:0/2726607506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42015d2d0 msgr2=0x7fb4200b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 --2- 172.21.15.46:0/2726607506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42015d2d0 0x7fb4200b8b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb4180097a0 tx=0x7fb41802b8d0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 -- 172.21.15.46:0/2726607506 shutdown_connections 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 --2- 172.21.15.46:0/2726607506 >> v2:172.21.15.121:3300/0 conn(0x7fb420160f70 0x7fb4200cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 --2- 172.21.15.46:0/2726607506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb42015d2d0 0x7fb4200b8b90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 -- 172.21.15.46:0/2726607506 >> 172.21.15.46:0/2726607506 conn(0x7fb420072230 msgr2=0x7fb4200cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 -- 172.21.15.46:0/2726607506 shutdown_connections 2023-12-16T21:16:11.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.527+0000 7fb424d43640 1 -- 172.21.15.46:0/2726607506 wait complete. 2023-12-16T21:16:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.528+0000 7fb424d43640 1 Processor -- start 2023-12-16T21:16:11.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.528+0000 7fb424d43640 1 -- start start 2023-12-16T21:16:11.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb424d43640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4200ce480 0x7fb420158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb424d43640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42015d2d0 0x7fb4200b5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb424d43640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb420160f70 0x7fb42015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.535 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb424d43640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42015a0c0 con 0x7fb4200ce480 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb424d43640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200cc5a0 con 0x7fb42015d2d0 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb424d43640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4201591d0 con 0x7fb420160f70 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb41e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4200ce480 0x7fb420158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb41e59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4200ce480 0x7fb420158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40952/0 (socket says 172.21.15.46:40952) 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb41e59c640 1 -- 172.21.15.46:0/3777139092 learned_addr learned my addr 172.21.15.46:0/3777139092 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb41ed9d640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb420160f70 0x7fb42015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.529+0000 7fb41dd9b640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42015d2d0 0x7fb4200b5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb41e59c640 1 -- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb420160f70 msgr2=0x7fb42015b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb41e59c640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb420160f70 0x7fb42015b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.536 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb41e59c640 1 -- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42015d2d0 msgr2=0x7fb4200b5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb41e59c640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb42015d2d0 0x7fb4200b5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb41e59c640 1 -- 172.21.15.46:0/3777139092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb418009400 con 0x7fb4200ce480 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb41e59c640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4200ce480 0x7fb420158660 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb41802bee0 tx=0x7fb4180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41803a6f0 con 0x7fb4200ce480 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41803a870 con 0x7fb4200ce480 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41803ac20 con 0x7fb4200ce480 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4201594b0 con 0x7fb4200ce480 2023-12-16T21:16:11.537 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.530+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb420159a70 con 0x7fb4200ce480 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.531+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4200b90d0 con 0x7fb4200ce480 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.531+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fb418030380 con 0x7fb4200ce480 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb40effd640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb40006ae40 0x7fb40006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb4180aab90 con 0x7fb4200ce480 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb40effd640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7fb400070ca0 0x7fb4000730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 --> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb400073770 con 0x7fb400070ca0 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4180af2c0 con 0x7fb4200ce480 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb41dd9b640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb40006ae40 0x7fb40006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.538 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.532+0000 7fb41ed9d640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7fb400070ca0 0x7fb4000730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.533+0000 7fb41dd9b640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb40006ae40 0x7fb40006d300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb4200cafd0 tx=0x7fb414009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.533+0000 7fb41ed9d640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7fb400070ca0 0x7fb4000730c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.539 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.533+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== osd.5 v2:172.21.15.121:6808/1440061954 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb400073770 con 0x7fb400070ca0 2023-12-16T21:16:11.546 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.547+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 --> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb4200ce8d0 con 0x7fb400070ca0 2023-12-16T21:16:11.547 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.547+0000 7fb40effd640 1 -- 172.21.15.46:0/3777139092 <== osd.5 v2:172.21.15.121:6808/1440061954 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb410001f00 con 0x7fb400070ca0 2023-12-16T21:16:11.547 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:11.547 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:11.548 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:11.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7fb400070ca0 msgr2=0x7fb4000730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6808/1440061954,v1:172.21.15.121:6809/1440061954] conn(0x7fb400070ca0 0x7fb4000730c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb40006ae40 msgr2=0x7fb40006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fb40006ae40 0x7fb40006d300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb4200cafd0 tx=0x7fb414009190 comp rx=0 tx=0).stop 2023-12-16T21:16:11.549 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4200ce480 msgr2=0x7fb420158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 --2- 172.21.15.46:0/3777139092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb4200ce480 0x7fb420158660 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fb41802bee0 tx=0x7fb4180046c0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb41ed9d640 1 -- 172.21.15.46:0/3777139092 reap_dead start 2023-12-16T21:16:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 shutdown_connections 2023-12-16T21:16:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 >> 172.21.15.46:0/3777139092 conn(0x7fb420072230 msgr2=0x7fb4200c6d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 shutdown_connections 2023-12-16T21:16:11.550 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.548+0000 7fb424d43640 1 -- 172.21.15.46:0/3777139092 wait complete. 2023-12-16T21:16:11.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-16T21:16:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.666+0000 7fe8909fc640 1 Processor -- start 2023-12-16T21:16:11.666 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.666+0000 7fe8909fc640 1 -- start start 2023-12-16T21:16:11.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe8909fc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe88c143490 0x7fe88c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe8909fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15e730 0x7fe88c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.670 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe8909fc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe88c15a4e0 0x7fe88c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe8909fc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88c15b520 con 0x7fe88c15a4e0 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe8909fc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88c143870 con 0x7fe88c143490 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe8909fc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15e2a0 con 0x7fe88c15e730 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe88c15a4e0 0x7fe88c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88ad9d640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fe88c15a4e0 0x7fe88c15a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40954/0 (socket says 172.21.15.46:40954) 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88ad9d640 1 -- 172.21.15.46:0/2154041778 learned_addr learned my addr 172.21.15.46:0/2154041778 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:11.671 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe889d9b640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15e730 0x7fe88c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88a59c640 1 --2- 172.21.15.46:0/2154041778 >> v2:172.21.15.121:3300/0 conn(0x7fe88c143490 0x7fe88c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88ad9d640 1 -- 172.21.15.46:0/2154041778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15e730 msgr2=0x7fe88c0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88ad9d640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15e730 0x7fe88c0b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.667+0000 7fe88ad9d640 1 -- 172.21.15.46:0/2154041778 >> v2:172.21.15.121:3300/0 conn(0x7fe88c143490 msgr2=0x7fe88c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88ad9d640 1 --2- 172.21.15.46:0/2154041778 >> v2:172.21.15.121:3300/0 conn(0x7fe88c143490 0x7fe88c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88ad9d640 1 -- 172.21.15.46:0/2154041778 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88c15b240 con 0x7fe88c15a4e0 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe889d9b640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15e730 0x7fe88c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:11.672 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88a59c640 1 --2- 172.21.15.46:0/2154041778 >> v2:172.21.15.121:3300/0 conn(0x7fe88c143490 0x7fe88c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:11.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88ad9d640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c15a4e0 0x7fe88c15a970 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe88c06e620 tx=0x7fe87c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126366f0ae692788 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88959a640 1 -- 172.21.15.46:0/2154041778 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c02ca70 con 0x7fe88c15a4e0 2023-12-16T21:16:11.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88959a640 1 -- 172.21.15.46:0/2154041778 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c02cbd0 con 0x7fe88c15a4e0 2023-12-16T21:16:11.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.668+0000 7fe88959a640 1 -- 172.21.15.46:0/2154041778 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0343f0 con 0x7fe88c15a4e0 2023-12-16T21:16:11.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 -- 172.21.15.46:0/2154041778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c15a4e0 msgr2=0x7fe88c15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.673 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c15a4e0 0x7fe88c15a970 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe88c06e620 tx=0x7fe87c02bed0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 -- 172.21.15.46:0/2154041778 shutdown_connections 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c15a4e0 0x7fe88c15a970 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 --2- 172.21.15.46:0/2154041778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15e730 0x7fe88c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 --2- 172.21.15.46:0/2154041778 >> v2:172.21.15.121:3300/0 conn(0x7fe88c143490 0x7fe88c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 -- 172.21.15.46:0/2154041778 >> 172.21.15.46:0/2154041778 conn(0x7fe88c06a4d0 msgr2=0x7fe88c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 -- 172.21.15.46:0/2154041778 shutdown_connections 2023-12-16T21:16:11.674 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.669+0000 7fe8909fc640 1 -- 172.21.15.46:0/2154041778 wait complete. 2023-12-16T21:16:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.670+0000 7fe8909fc640 1 Processor -- start 2023-12-16T21:16:11.675 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.670+0000 7fe8909fc640 1 -- start start 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe8909fc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 0x7fe88c138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe8909fc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15a4e0 0x7fe88c13b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe8909fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe88c15e730 0x7fe88c13b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe8909fc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15d0a0 con 0x7fe88c143490 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe8909fc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c1394e0 con 0x7fe88c15e730 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe8909fc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88c15c420 con 0x7fe88c15a4e0 2023-12-16T21:16:11.678 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 0x7fe88c138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe88a59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 0x7fe88c138fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40970/0 (socket says 172.21.15.46:40970) 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe88a59c640 1 -- 172.21.15.46:0/329897096 learned_addr learned my addr 172.21.15.46:0/329897096 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe889d9b640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15a4e0 0x7fe88c13b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.671+0000 7fe88ad9d640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe88c15e730 0x7fe88c13b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe88a59c640 1 -- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15a4e0 msgr2=0x7fe88c13b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe88a59c640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15a4e0 0x7fe88c13b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.679 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe88a59c640 1 -- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe88c15e730 msgr2=0x7fe88c13b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe88a59c640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe88c15e730 0x7fe88c13b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe88a59c640 1 -- 172.21.15.46:0/329897096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009b30 con 0x7fe88c143490 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe88a59c640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 0x7fe88c138fa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c7f0 tx=0x7fe88400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884007a30 con 0x7fe88c143490 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe884007b90 con 0x7fe88c143490 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe88c15c700 con 0x7fe88c143490 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88400ce40 con 0x7fe88c143490 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.672+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe88c15ccc0 con 0x7fe88c143490 2023-12-16T21:16:11.680 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.673+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88c15d1e0 con 0x7fe88c143490 2023-12-16T21:16:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.673+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fe8840113f0 con 0x7fe88c143490 2023-12-16T21:16:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.674+0000 7fe882ffd640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe86c06ae70 0x7fe86c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.674+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe884089470 con 0x7fe88c143490 2023-12-16T21:16:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.674+0000 7fe889d9b640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe86c06ae70 0x7fe86c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.674+0000 7fe889d9b640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe86c06ae70 0x7fe86c06d330 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0b42c0 tx=0x7fe874008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.681 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.678+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fe88c15d1e0 con 0x7fe88c143490 2023-12-16T21:16:11.855 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.855+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe88c15aeb0 con 0x7fe86c06ae70 2023-12-16T21:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.856+0000 7fe882ffd640 1 -- 172.21.15.46:0/329897096 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe87400c050 con 0x7fe86c06ae70 2023-12-16T21:16:11.856 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 5 to deep-scrub 2023-12-16T21:16:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe86c06ae70 msgr2=0x7fe86c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe86c06ae70 0x7fe86c06d330 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0b42c0 tx=0x7fe874008040 comp rx=0 tx=0).stop 2023-12-16T21:16:11.858 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 msgr2=0x7fe88c138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 0x7fe88c138fa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe88400c7f0 tx=0x7fe88400cbf0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 shutdown_connections 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fe86c06ae70 0x7fe86c06d330 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe88c15e730 0x7fe88c13b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe88c15a4e0 0x7fe88c13b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 --2- 172.21.15.46:0/329897096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe88c143490 0x7fe88c138fa0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 >> 172.21.15.46:0/329897096 conn(0x7fe88c06a4d0 msgr2=0x7fe88c070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 shutdown_connections 2023-12-16T21:16:11.859 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.858+0000 7fe8909fc640 1 -- 172.21.15.46:0/329897096 wait complete. 2023-12-16T21:16:11.869 INFO:tasks.ceph:Scrubbing osd.6 2023-12-16T21:16:11.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.979+0000 7f70c4dd7640 1 Processor -- start 2023-12-16T21:16:11.979 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.979+0000 7f70c4dd7640 1 -- start start 2023-12-16T21:16:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70c4dd7640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70c00b1400 0x7f70c015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70c4dd7640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c015e2a0 con 0x7f70c015e730 2023-12-16T21:16:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70c4dd7640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c015b240 con 0x7f70c0143490 2023-12-16T21:16:11.983 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70c4dd7640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c015b520 con 0x7f70c00b1400 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70bed9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f70c015e730 0x7f70c015aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50420/0 (socket says 172.21.15.46:50420) 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70bed9d640 1 -- 172.21.15.46:0/1722164420 learned_addr learned my addr 172.21.15.46:0/1722164420 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70bdd9b640 1 --2- 172.21.15.46:0/1722164420 >> v2:172.21.15.121:3300/0 conn(0x7f70c00b1400 0x7f70c015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907821262 0 0) 0x7f70c015e2a0 con 0x7f70c015e730 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.980+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70ac003520 con 0x7f70c015e730 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387238095 0 0) 0x7f70ac003520 con 0x7f70c015e730 2023-12-16T21:16:11.984 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 >> v2:172.21.15.121:3300/0 conn(0x7f70c00b1400 msgr2=0x7f70c015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 --2- 172.21.15.46:0/1722164420 >> v2:172.21.15.121:3300/0 conn(0x7f70c00b1400 0x7f70c015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 >> v1:172.21.15.121:6789/0 conn(0x7f70c0143490 legacy=0x7f70c00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c015dc50 con 0x7f70c015e730 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f70b00019f0 con 0x7f70c015e730 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 >> v1:172.21.15.46:6789/0 conn(0x7f70c015e730 legacy=0x7f70c015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70ac003970 0x7f70ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.985 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70ac0047c0 0x7f70ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c015b520 con 0x7f70ac005180 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac00c020 con 0x7f70ac0047c0 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac00c1e0 con 0x7f70ac003970 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70be59c640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70ac003970 0x7f70ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.986 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bdd9b640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70ac0047c0 0x7f70ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bed9d640 1 -- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70ac003970 msgr2=0x7f70ac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70ac003970 0x7f70ac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bed9d640 1 -- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70ac0047c0 msgr2=0x7f70ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70ac0047c0 0x7f70ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.981+0000 7f70bed9d640 1 -- 172.21.15.46:0/1722164420 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac003520 con 0x7f70ac005180 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.982+0000 7f70bed9d640 1 -- 172.21.15.46:0/1722164420 reap_dead start 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.982+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70ac00b980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f70b000a830 tx=0x7f70ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe70467500a9b92 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.982+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b00072d0 con 0x7f70ac005180 2023-12-16T21:16:11.987 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.982+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b002f070 con 0x7f70ac005180 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.982+0000 7f70bd59a640 1 -- 172.21.15.46:0/1722164420 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b0007690 con 0x7f70ac005180 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1722164420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 msgr2=0x7f70ac00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70ac00b980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f70b000a830 tx=0x7f70ac0057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1722164420 shutdown_connections 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 --2- 172.21.15.46:0/1722164420 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70ac00b980 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1722164420 >> 172.21.15.46:0/1722164420 conn(0x7f70c006a4d0 msgr2=0x7f70c006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1722164420 shutdown_connections 2023-12-16T21:16:11.988 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.983+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1722164420 wait complete. 2023-12-16T21:16:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.984+0000 7f70c4dd7640 1 Processor -- start 2023-12-16T21:16:11.991 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.984+0000 7f70c4dd7640 1 -- start start 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70c4dd7640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70c01413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70c4dd7640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70c013fe50 0x7f70c013b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70c4dd7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c013ecd0 0x7f70c013ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70c4dd7640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c01418f0 con 0x7f70ac005180 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70c4dd7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c0140310 con 0x7f70c013ecd0 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70c4dd7640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c01405f0 con 0x7f70c013fe50 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70c01413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.992 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70c01413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40990/0 (socket says 172.21.15.46:40990) 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 -- 172.21.15.46:0/1961276613 learned_addr learned my addr 172.21.15.46:0/1961276613 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70bdd9b640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70c013fe50 0x7f70c013b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c013ecd0 0x7f70c013ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 -- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70c013fe50 msgr2=0x7f70c013b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70c013fe50 0x7f70c013b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 -- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c013ecd0 msgr2=0x7f70c013ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70c013ecd0 0x7f70c013ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:11.993 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70be59c640 1 -- 172.21.15.46:0/1961276613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c01408d0 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.985+0000 7f70bdd9b640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70c013fe50 0x7f70c013b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.986+0000 7f70be59c640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70c01413b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f70b800f5f0 tx=0x7f70b800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.986+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b80052e0 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.986+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b8005440 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.986+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b8003810 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.986+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c0140bb0 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.986+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c013bd30 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.987+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70c013bef0 con 0x7f70ac005180 2023-12-16T21:16:11.994 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.987+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f70b8003970 con 0x7f70ac005180 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70b6ffd640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70a006ae40 0x7f70a006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f70b80891a0 con 0x7f70ac005180 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70b6ffd640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7f70a0070ca0 0x7f70a00730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 --> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70a0073770 con 0x7f70a0070ca0 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70b808b860 con 0x7f70ac005180 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70bdd9b640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70a006ae40 0x7f70a006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7f70a0070ca0 0x7f70a00730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70bdd9b640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70a006ae40 0x7f70a006d300 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f70c00b42e0 tx=0x7f70a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.988+0000 7f70bed9d640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7f70a0070ca0 0x7f70a00730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:11.995 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:11.989+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== osd.6 v2:172.21.15.121:6814/3722378128 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f70a0073770 con 0x7f70a0070ca0 2023-12-16T21:16:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.002+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 --> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f70c0069400 con 0x7f70a0070ca0 2023-12-16T21:16:12.002 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70b6ffd640 1 -- 172.21.15.46:0/1961276613 <== osd.6 v2:172.21.15.121:6814/3722378128 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f70c0069400 con 0x7f70a0070ca0 2023-12-16T21:16:12.003 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:12.003 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7f70a0070ca0 msgr2=0x7f70a00730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6814/3722378128,v1:172.21.15.121:6815/3722378128] conn(0x7f70a0070ca0 0x7f70a00730c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.004 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70a006ae40 msgr2=0x7f70a006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f70a006ae40 0x7f70a006d300 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f70c00b42e0 tx=0x7f70a8008040 comp rx=0 tx=0).stop 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 msgr2=0x7f70c01413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 --2- 172.21.15.46:0/1961276613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f70ac005180 0x7f70c01413b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f70b800f5f0 tx=0x7f70b800fab0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70bed9d640 1 -- 172.21.15.46:0/1961276613 reap_dead start 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 shutdown_connections 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 >> 172.21.15.46:0/1961276613 conn(0x7f70c006a4d0 msgr2=0x7f70c0070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.005 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 shutdown_connections 2023-12-16T21:16:12.006 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.003+0000 7f70c4dd7640 1 -- 172.21.15.46:0/1961276613 wait complete. 2023-12-16T21:16:12.013 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-16T21:16:12.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.121+0000 7f7f301a9640 1 Processor -- start 2023-12-16T21:16:12.121 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.121+0000 7f7f301a9640 1 -- start start 2023-12-16T21:16:12.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f301a9640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f28143490 0x7f7f280b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f301a9640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7f2815e730 0x7f7f280b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f301a9640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2815e2a0 con 0x7f7f2815a4e0 2023-12-16T21:16:12.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f301a9640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2815b520 con 0x7f7f28143490 2023-12-16T21:16:12.123 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f301a9640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f28143870 con 0x7f7f2815e730 2023-12-16T21:16:12.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f28143490 0x7f7f280b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f7f28143490 0x7f7f280b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:40996/0 (socket says 172.21.15.46:40996) 2023-12-16T21:16:12.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 -- 172.21.15.46:0/3683871044 learned_addr learned my addr 172.21.15.46:0/3683871044 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:12.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2d744640 1 --2- 172.21.15.46:0/3683871044 >> v2:172.21.15.121:3300/0 conn(0x7f7f2815e730 0x7f7f280b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 -- 172.21.15.46:0/3683871044 >> v2:172.21.15.121:3300/0 conn(0x7f7f2815e730 msgr2=0x7f7f280b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:12.124 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 --2- 172.21.15.46:0/3683871044 >> v2:172.21.15.121:3300/0 conn(0x7f7f2815e730 0x7f7f280b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 -- 172.21.15.46:0/3683871044 >> v1:172.21.15.121:6789/0 conn(0x7f7f2815a4e0 legacy=0x7f7f2815a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2df45640 1 -- 172.21.15.46:0/3683871044 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2815b240 con 0x7f7f28143490 2023-12-16T21:16:12.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.122+0000 7f7f2d744640 1 --2- 172.21.15.46:0/3683871044 >> v2:172.21.15.121:3300/0 conn(0x7f7f2815e730 0x7f7f280b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:12.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.123+0000 7f7f2df45640 1 --2- 172.21.15.46:0/3683871044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f28143490 0x7f7f280b0cf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7f280aeca0 tx=0x7f7f2402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634c1c4da67d8500 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.123+0000 7f7f2cf43640 1 -- 172.21.15.46:0/3683871044 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24030860 con 0x7f7f28143490 2023-12-16T21:16:12.125 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.123+0000 7f7f2cf43640 1 -- 172.21.15.46:0/3683871044 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f240309c0 con 0x7f7f28143490 2023-12-16T21:16:12.126 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.123+0000 7f7f2cf43640 1 -- 172.21.15.46:0/3683871044 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f24030d50 con 0x7f7f28143490 2023-12-16T21:16:12.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 -- 172.21.15.46:0/3683871044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f28143490 msgr2=0x7f7f280b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.129 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3683871044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f28143490 0x7f7f280b0cf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7f280aeca0 tx=0x7f7f2402b9a0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 -- 172.21.15.46:0/3683871044 shutdown_connections 2023-12-16T21:16:12.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3683871044 >> v2:172.21.15.121:3300/0 conn(0x7f7f2815e730 0x7f7f280b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3683871044 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f28143490 0x7f7f280b0cf0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.130 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 -- 172.21.15.46:0/3683871044 >> 172.21.15.46:0/3683871044 conn(0x7f7f2806a4d0 msgr2=0x7f7f2806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 -- 172.21.15.46:0/3683871044 shutdown_connections 2023-12-16T21:16:12.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.124+0000 7f7f301a9640 1 -- 172.21.15.46:0/3683871044 wait complete. 2023-12-16T21:16:12.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.125+0000 7f7f301a9640 1 Processor -- start 2023-12-16T21:16:12.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.125+0000 7f7f301a9640 1 -- start start 2023-12-16T21:16:12.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f28143490 0x7f7f281f4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.131 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 0x7f7f281f49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f2815e730 0x7f7f281f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28140030 con 0x7f7f2815a4e0 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2813b320 con 0x7f7f2815e730 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2813b5c0 con 0x7f7f28143490 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 0x7f7f281f49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 0x7f7f281f49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41002/0 (socket says 172.21.15.46:41002) 2023-12-16T21:16:12.132 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 -- 172.21.15.46:0/3324517108 learned_addr learned my addr 172.21.15.46:0/3324517108 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2e746640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f2815e730 0x7f7f281f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2df45640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f28143490 0x7f7f281f4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 -- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f28143490 msgr2=0x7f7f281f4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f28143490 0x7f7f281f4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 -- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f2815e730 msgr2=0x7f7f281f4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f2815e730 0x7f7f281f4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.133 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 -- 172.21.15.46:0/3324517108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240094d0 con 0x7f7f2815a4e0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f2d744640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 0x7f7f281f49d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d7d0 tx=0x7f7f1800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18004030 con 0x7f7f2815a4e0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f18004190 con 0x7f7f2815a4e0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f28140280 con 0x7f7f2815a4e0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18004520 con 0x7f7f2815a4e0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.126+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f28159a50 con 0x7f7f2815a4e0 2023-12-16T21:16:12.134 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.127+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28159cd0 con 0x7f7f2815a4e0 2023-12-16T21:16:12.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.128+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f7f1800b6c0 con 0x7f7f2815a4e0 2023-12-16T21:16:12.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.128+0000 7f7f1e7fc640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7f0c06ae40 0x7f7f0c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.128+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7f18089430 con 0x7f7f2815a4e0 2023-12-16T21:16:12.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.128+0000 7f7f2df45640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7f0c06ae40 0x7f7f0c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.135 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.129+0000 7f7f2df45640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7f0c06ae40 0x7f7f0c06d300 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7f24036000 tx=0x7f7f24008030 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.136 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.132+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f7f18054870 con 0x7f7f2815a4e0 2023-12-16T21:16:12.307 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.308+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f2815df30 con 0x7f7f0c06ae40 2023-12-16T21:16:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.308+0000 7f7f1e7fc640 1 -- 172.21.15.46:0/3324517108 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7f24030bb0 con 0x7f7f0c06ae40 2023-12-16T21:16:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 6 to deep-scrub 2023-12-16T21:16:12.310 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7f0c06ae40 msgr2=0x7f7f0c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7f0c06ae40 0x7f7f0c06d300 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7f24036000 tx=0x7f7f24008030 comp rx=0 tx=0).stop 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 msgr2=0x7f7f281f49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 0x7f7f281f49d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800d7d0 tx=0x7f7f1800dc90 comp rx=0 tx=0).stop 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 shutdown_connections 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f7f0c06ae40 0x7f7f0c06d300 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7f2815a4e0 0x7f7f281f49d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.311 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f2815e730 0x7f7f281f4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 --2- 172.21.15.46:0/3324517108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f28143490 0x7f7f281f4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 >> 172.21.15.46:0/3324517108 conn(0x7f7f2806a4d0 msgr2=0x7f7f2815a930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 shutdown_connections 2023-12-16T21:16:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.310+0000 7f7f301a9640 1 -- 172.21.15.46:0/3324517108 wait complete. 2023-12-16T21:16:12.321 INFO:tasks.ceph:Scrubbing osd.7 2023-12-16T21:16:12.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-16T21:16:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.431+0000 7f73c8cb4640 1 Processor -- start 2023-12-16T21:16:12.431 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.431+0000 7f73c8cb4640 1 -- start start 2023-12-16T21:16:12.432 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c8cb4640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73c415d330 0x7f73c40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c8cb4640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40b2b10 con 0x7f73c40d3900 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c8cb4640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c40d1590 con 0x7f73c40d2910 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c8cb4640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73c40d1870 con 0x7f73c415d330 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c259c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f73c415d330 0x7f73c40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c2d9d640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f73c40d3900 0x7f73c40b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50436/0 (socket says 172.21.15.46:50436) 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.432+0000 7f73c2d9d640 1 -- 172.21.15.46:0/3072943428 learned_addr learned my addr 172.21.15.46:0/3072943428 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.433+0000 7f73c259c640 1 -- 172.21.15.46:0/3072943428 >> v1:172.21.15.121:6789/0 conn(0x7f73c40d2910 legacy=0x7f73c40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.433+0000 7f73c259c640 1 -- 172.21.15.46:0/3072943428 >> v1:172.21.15.46:6789/0 conn(0x7f73c40d3900 legacy=0x7f73c40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.433 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.433+0000 7f73c159a640 1 -- 172.21.15.46:0/3072943428 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216537317 0 0) 0x7f73c40b2b10 con 0x7f73c40d3900 2023-12-16T21:16:12.434 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.433+0000 7f73c259c640 1 -- 172.21.15.46:0/3072943428 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c40d1b50 con 0x7f73c415d330 2023-12-16T21:16:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.433+0000 7f73c259c640 1 --2- 172.21.15.46:0/3072943428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c415d330 0x7f73c40b8b80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0097a0 tx=0x7f73bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f8407056bd40cb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c159a640 1 -- 172.21.15.46:0/3072943428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc02e070 con 0x7f73c415d330 2023-12-16T21:16:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c159a640 1 -- 172.21.15.46:0/3072943428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc0315a0 con 0x7f73c415d330 2023-12-16T21:16:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c159a640 1 -- 172.21.15.46:0/3072943428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc031930 con 0x7f73c415d330 2023-12-16T21:16:12.435 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3072943428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c415d330 msgr2=0x7f73c40b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 --2- 172.21.15.46:0/3072943428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c415d330 0x7f73c40b8b80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0097a0 tx=0x7f73bc02b840 comp rx=0 tx=0).stop 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3072943428 shutdown_connections 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 --2- 172.21.15.46:0/3072943428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c415d330 0x7f73c40b8b80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3072943428 >> 172.21.15.46:0/3072943428 conn(0x7f73c4072020 msgr2=0x7f73c4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3072943428 shutdown_connections 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.434+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3072943428 wait complete. 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.435+0000 7f73c8cb4640 1 Processor -- start 2023-12-16T21:16:12.436 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.435+0000 7f73c8cb4640 1 -- start start 2023-12-16T21:16:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c8cb4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c40d2910 0x7f73c40c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.439 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c8cb4640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c40d3900 0x7f73c415f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c8cb4640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c415d330 0x7f73c415f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c8cb4640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c40c7f20 con 0x7f73c40d3900 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c8cb4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c40c72a0 con 0x7f73c40d2910 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c8cb4640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73c40c7580 con 0x7f73c415d330 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c40d3900 0x7f73c415f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c1d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c40d3900 0x7f73c415f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41010/0 (socket says 172.21.15.46:41010) 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c1d9b640 1 -- 172.21.15.46:0/3627940291 learned_addr learned my addr 172.21.15.46:0/3627940291 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c259c640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c40d2910 0x7f73c40c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.440 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.436+0000 7f73c2d9d640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c415d330 0x7f73c415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c1d9b640 1 -- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c415d330 msgr2=0x7f73c415f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c1d9b640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c415d330 0x7f73c415f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c1d9b640 1 -- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c40d2910 msgr2=0x7f73c40c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c1d9b640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c40d2910 0x7f73c40c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c1d9b640 1 -- 172.21.15.46:0/3627940291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc009400 con 0x7f73c40d3900 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c259c640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73c40d2910 0x7f73c40c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c2d9d640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73c415d330 0x7f73c415f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c1d9b640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c40d3900 0x7f73c415f440 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c430 tx=0x7f73ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.441 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac011790 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ac0118f0 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c40c7860 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ac011ca0 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.437+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c4160250 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.438+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f73c40c9ff0 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.438+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f73ac01c460 con 0x7f73c40d3900 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73baffd640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73a406ae40 0x7f73a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.442 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f73ac0892b0 con 0x7f73c40d3900 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73baffd640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7f73a4070ca0 0x7f73a40730c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 --> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73a4073770 con 0x7f73a4070ca0 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73ac08b970 con 0x7f73c40d3900 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73c259c640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73a406ae40 0x7f73a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.439+0000 7f73c2d9d640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7f73a4070ca0 0x7f73a40730c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.440+0000 7f73c259c640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73a406ae40 0x7f73a406d300 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f73bc036000 tx=0x7f73bc03d040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.443 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.440+0000 7f73c2d9d640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7f73a4070ca0 0x7f73a40730c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.444 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.440+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== osd.7 v2:172.21.15.121:6800/2871933776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f73a4073770 con 0x7f73a4070ca0 2023-12-16T21:16:12.453 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.453+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 --> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f73c40c82a0 con 0x7f73a4070ca0 2023-12-16T21:16:12.454 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.454+0000 7f73baffd640 1 -- 172.21.15.46:0/3627940291 <== osd.7 v2:172.21.15.121:6800/2871933776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f73c40c82a0 con 0x7f73a4070ca0 2023-12-16T21:16:12.454 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2023-12-16T21:16:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-16T21:16:12.454 INFO:teuthology.orchestra.run.smithi046.stdout:} 2023-12-16T21:16:12.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7f73a4070ca0 msgr2=0x7f73a40730c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.455 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6800/2871933776,v1:172.21.15.121:6801/2871933776] conn(0x7f73a4070ca0 0x7f73a40730c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73a406ae40 msgr2=0x7f73a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f73a406ae40 0x7f73a406d300 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f73bc036000 tx=0x7f73bc03d040 comp rx=0 tx=0).stop 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c40d3900 msgr2=0x7f73c415f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 --2- 172.21.15.46:0/3627940291 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f73c40d3900 0x7f73c415f440 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f73ac00c430 tx=0x7f73ac00c8f0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c2d9d640 1 -- 172.21.15.46:0/3627940291 reap_dead start 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 shutdown_connections 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 >> 172.21.15.46:0/3627940291 conn(0x7f73c4072020 msgr2=0x7f73c4072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 shutdown_connections 2023-12-16T21:16:12.456 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.455+0000 7f73c8cb4640 1 -- 172.21.15.46:0/3627940291 wait complete. 2023-12-16T21:16:12.465 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-16T21:16:12.572 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.573+0000 7f84fbc4f640 1 Processor -- start 2023-12-16T21:16:12.573 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.573+0000 7f84fbc4f640 1 -- start start 2023-12-16T21:16:12.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fbc4f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f4155790 0x7f84f406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.574 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fbc4f640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84f415b5e0 0x7f84f415ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fbc4f640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f4155b70 con 0x7f84f406a4d0 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fbc4f640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f406ee30 con 0x7f84f415b5e0 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fbc4f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f406f3b0 con 0x7f84f4155790 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f91ea640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f84f415b5e0 0x7f84f415ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fa1ec640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f84f406a4d0 0x7f84f40714a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50440/0 (socket says 172.21.15.46:50440) 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84fa1ec640 1 -- 172.21.15.46:0/2649553862 learned_addr learned my addr 172.21.15.46:0/2649553862 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f99eb640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f4155790 0x7f84f406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f91ea640 1 -- 172.21.15.46:0/2649553862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f4155790 msgr2=0x7f84f406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:12.575 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f91ea640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f4155790 0x7f84f406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f91ea640 1 -- 172.21.15.46:0/2649553862 >> v1:172.21.15.46:6789/0 conn(0x7f84f406a4d0 legacy=0x7f84f40714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f91ea640 1 -- 172.21.15.46:0/2649553862 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f415c170 con 0x7f84f415b5e0 2023-12-16T21:16:12.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f89e9640 1 -- 172.21.15.46:0/2649553862 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191303802 0 0) 0x7f84f4155b70 con 0x7f84f406a4d0 2023-12-16T21:16:12.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.574+0000 7f84f99eb640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f4155790 0x7f84f406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:16:12.576 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.575+0000 7f84f91ea640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f415b5e0 0x7f84f415ba60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f84e40097a0 tx=0x7f84e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c231b1df907855e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.580 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.575+0000 7f84f89e9640 1 -- 172.21.15.46:0/2649553862 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4031830 con 0x7f84f415b5e0 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.575+0000 7f84f89e9640 1 -- 172.21.15.46:0/2649553862 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e4031990 con 0x7f84f415b5e0 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.575+0000 7f84f89e9640 1 -- 172.21.15.46:0/2649553862 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4031d20 con 0x7f84f415b5e0 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 -- 172.21.15.46:0/2649553862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f415b5e0 msgr2=0x7f84f415ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f415b5e0 0x7f84f415ba60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f84e40097a0 tx=0x7f84e402b8d0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 -- 172.21.15.46:0/2649553862 shutdown_connections 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f415b5e0 0x7f84f415ba60 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.581 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/2649553862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f4155790 0x7f84f406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 -- 172.21.15.46:0/2649553862 >> 172.21.15.46:0/2649553862 conn(0x7f84f40b1e50 msgr2=0x7f84f40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 -- 172.21.15.46:0/2649553862 shutdown_connections 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.576+0000 7f84fbc4f640 1 -- 172.21.15.46:0/2649553862 wait complete. 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.577+0000 7f84fbc4f640 1 Processor -- start 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.577+0000 7f84fbc4f640 1 -- start start 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 0x7f84f414d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f4155790 0x7f84f41582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.582 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f415b5e0 0x7f84f41587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f415adb0 con 0x7f84f406a4d0 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4159d10 con 0x7f84f4155790 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4159ff0 con 0x7f84f415b5e0 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 0x7f84f414d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 0x7f84f414d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41024/0 (socket says 172.21.15.46:41024) 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 -- 172.21.15.46:0/77250729 learned_addr learned my addr 172.21.15.46:0/77250729 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f91ea640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f4155790 0x7f84f41582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.583 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fa1ec640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f415b5e0 0x7f84f41587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 -- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f415b5e0 msgr2=0x7f84f41587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f415b5e0 0x7f84f41587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 -- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f4155790 msgr2=0x7f84f41582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f4155790 0x7f84f41582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 -- 172.21.15.46:0/77250729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4009400 con 0x7f84f406a4d0 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f91ea640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f4155790 0x7f84f41582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:12.584 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fa1ec640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f415b5e0 0x7f84f41587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84f99eb640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 0x7f84f414d550 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f84f000c7f0 tx=0x7f84f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0007820 con 0x7f84f406a4d0 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0007980 con 0x7f84f406a4d0 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.578+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f415a2d0 con 0x7f84f406a4d0 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.579+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f0007cf0 con 0x7f84f406a4d0 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.579+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f415a890 con 0x7f84f406a4d0 2023-12-16T21:16:12.585 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.579+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f4069400 con 0x7f84f406a4d0 2023-12-16T21:16:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.580+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f84f00113f0 con 0x7f84f406a4d0 2023-12-16T21:16:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.580+0000 7f84ea7fc640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f84d806ae70 0x7f84d806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.580+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f84f0089600 con 0x7f84f406a4d0 2023-12-16T21:16:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.580+0000 7f84f91ea640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f84d806ae70 0x7f84d806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.581+0000 7f84f91ea640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f84d806ae70 0x7f84d806d330 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f84e4036000 tx=0x7f84e403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.586 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.584+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f84f00549b0 con 0x7f84f406a4d0 2023-12-16T21:16:12.760 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.760+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f84f40cfdf0 con 0x7f84d806ae70 2023-12-16T21:16:12.761 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.761+0000 7f84ea7fc640 1 -- 172.21.15.46:0/77250729 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f84e40317d0 con 0x7f84d806ae70 2023-12-16T21:16:12.761 INFO:teuthology.orchestra.run.smithi046.stderr:instructed osd(s) 7 to deep-scrub 2023-12-16T21:16:12.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f84d806ae70 msgr2=0x7f84d806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f84d806ae70 0x7f84d806d330 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f84e4036000 tx=0x7f84e403c040 comp rx=0 tx=0).stop 2023-12-16T21:16:12.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 msgr2=0x7f84f414d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 0x7f84f414d550 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f84f000c7f0 tx=0x7f84f000ccb0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.763 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 shutdown_connections 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f84d806ae70 0x7f84d806d330 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f415b5e0 0x7f84f41587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f4155790 0x7f84f41582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 --2- 172.21.15.46:0/77250729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84f406a4d0 0x7f84f414d550 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 >> 172.21.15.46:0/77250729 conn(0x7f84f40b1e50 msgr2=0x7f84f40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 shutdown_connections 2023-12-16T21:16:12.764 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.763+0000 7f84fbc4f640 1 -- 172.21.15.46:0/77250729 wait complete. 2023-12-16T21:16:12.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:16:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.880+0000 7f1277e5c640 1 Processor -- start 2023-12-16T21:16:12.880 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.880+0000 7f1277e5c640 1 -- start start 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1277e5c640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f127006bbf0 0x7f12700b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1277e5c640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12700b2eb0 con 0x7f12701415f0 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1277e5c640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12700b3190 con 0x7f1270144a00 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1277e5c640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12700b3470 con 0x7f127006bbf0 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1275bf8640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f12701415f0 0x7f1270141e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:50450/0 (socket says 172.21.15.46:50450) 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1275bf8640 1 -- 172.21.15.46:0/2229800137 learned_addr learned my addr 172.21.15.46:0/2229800137 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f12763f9640 1 --2- 172.21.15.46:0/2229800137 >> v2:172.21.15.121:3300/0 conn(0x7f127006bbf0 0x7f12700b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370815941 0 0) 0x7f12700b2eb0 con 0x7f12701415f0 2023-12-16T21:16:12.884 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 --> v1:172.21.15.46:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f125c003520 con 0x7f12701415f0 2023-12-16T21:16:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 <== mon.0 v1:172.21.15.46:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 338693965 0 0) 0x7f125c003520 con 0x7f12701415f0 2023-12-16T21:16:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 >> v2:172.21.15.121:3300/0 conn(0x7f127006bbf0 msgr2=0x7f12700b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1274bf6640 1 --2- 172.21.15.46:0/2229800137 >> v2:172.21.15.121:3300/0 conn(0x7f127006bbf0 0x7f12700b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.881+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 >> v1:172.21.15.121:6789/0 conn(0x7f1270144a00 legacy=0x7f127006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 --> v1:172.21.15.46:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12700b3750 con 0x7f12701415f0 2023-12-16T21:16:12.885 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 <== mon.0 v1:172.21.15.46:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1913365701 0 0) 0x7f126c001670 con 0x7f12701415f0 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 >> v1:172.21.15.46:6789/0 conn(0x7f12701415f0 legacy=0x7f1270141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c003970 0x7f125c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f125c0047c0 0x7f125c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125c005180 0x7f125c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700b3470 con 0x7f125c005180 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125c00c020 con 0x7f125c0047c0 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125c00c1e0 con 0x7f125c003970 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12763f9640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125c005180 0x7f125c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.886 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12753f7640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f125c0047c0 0x7f125c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f1275bf8640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c003970 0x7f125c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12763f9640 1 -- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c003970 msgr2=0x7f125c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12763f9640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c003970 0x7f125c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12763f9640 1 -- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f125c0047c0 msgr2=0x7f125c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12763f9640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f125c0047c0 0x7f125c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.882+0000 7f12763f9640 1 -- 172.21.15.46:0/2229800137 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f125c003520 con 0x7f125c005180 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f12763f9640 1 -- 172.21.15.46:0/2229800137 reap_dead start 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f12763f9640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125c005180 0x7f125c00b980 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1268009fc0 tx=0x7f125c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89969fd959589866 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.887 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126802cc60 con 0x7f125c005180 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126800f040 con 0x7f125c005180 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f1274bf6640 1 -- 172.21.15.46:0/2229800137 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268003990 con 0x7f125c005180 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f1277e5c640 1 -- 172.21.15.46:0/2229800137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125c005180 msgr2=0x7f125c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.883+0000 7f1277e5c640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125c005180 0x7f125c00b980 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1268009fc0 tx=0x7f125c0057f0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.884+0000 7f1277e5c640 1 -- 172.21.15.46:0/2229800137 shutdown_connections 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.884+0000 7f1277e5c640 1 --2- 172.21.15.46:0/2229800137 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f125c005180 0x7f125c00b980 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.884+0000 7f1277e5c640 1 -- 172.21.15.46:0/2229800137 >> 172.21.15.46:0/2229800137 conn(0x7f1270070e70 msgr2=0x7f1270071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.884+0000 7f1277e5c640 1 -- 172.21.15.46:0/2229800137 shutdown_connections 2023-12-16T21:16:12.888 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.884+0000 7f1277e5c640 1 -- 172.21.15.46:0/2229800137 wait complete. 2023-12-16T21:16:12.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.885+0000 7f1277e5c640 1 Processor -- start 2023-12-16T21:16:12.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.885+0000 7f1277e5c640 1 -- start start 2023-12-16T21:16:12.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.885+0000 7f1277e5c640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c005180 0x7f12700cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.885+0000 7f1277e5c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12701f6f90 0x7f12701f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1277e5c640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 0x7f1270158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.891 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1277e5c640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12700c9bc0 con 0x7f12701f3cb0 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1277e5c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12701f32e0 con 0x7f12701f6f90 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1277e5c640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12700ccce0 con 0x7f125c005180 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 0x7f1270158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 0x7f1270158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41032/0 (socket says 172.21.15.46:41032) 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 -- 172.21.15.46:0/890738210 learned_addr learned my addr 172.21.15.46:0/890738210 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12753f7640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12701f6f90 0x7f12701f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1275bf8640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c005180 0x7f12700cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 -- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c005180 msgr2=0x7f12700cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.892 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c005180 0x7f12700cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 -- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12701f6f90 msgr2=0x7f12701f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12701f6f90 0x7f12701f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 -- 172.21.15.46:0/890738210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12701f2f90 con 0x7f12701f3cb0 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12763f9640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 0x7f1270158730 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f126802c6d0 tx=0x7f1268009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268035d40 con 0x7f12701f3cb0 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1268035ec0 con 0x7f12701f3cb0 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12700c9320 con 0x7f12701f3cb0 2023-12-16T21:16:12.893 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1268003bd0 con 0x7f12701f3cb0 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.886+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12700c98e0 con 0x7f12701f3cb0 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.887+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12701f3490 con 0x7f12701f3cb0 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.888+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f1268003d30 con 0x7f12701f3cb0 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.888+0000 7f12667fc640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f125406ae70 0x7f125406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.888+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f12680abe20 con 0x7f12701f3cb0 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.889+0000 7f1275bf8640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f125406ae70 0x7f125406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:12.894 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.889+0000 7f1275bf8640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f125406ae70 0x7f125406d330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f126c0017a0 tx=0x7f126c001610 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:12.895 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:12.893+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f12701f3490 con 0x7f12701f3cb0 2023-12-16T21:16:13.065 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.065+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1270144440 con 0x7f125406ae70 2023-12-16T21:16:13.067 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.068+0000 7f12667fc640 1 -- 172.21.15.46:0/890738210 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51029 (secure 0 0 0) 0x7f126c004c70 con 0x7f125406ae70 2023-12-16T21:16:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:13.068 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:16:13.069 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.069+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f125406ae70 msgr2=0x7f125406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.069+0000 7f1277e5c640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f125406ae70 0x7f125406d330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f126c0017a0 tx=0x7f126c001610 comp rx=0 tx=0).stop 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.069+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 msgr2=0x7f1270158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.069+0000 7f1277e5c640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 0x7f1270158730 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f126802c6d0 tx=0x7f1268009750 comp rx=0 tx=0).stop 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.069+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 shutdown_connections 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f125406ae70 0x7f125406d330 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12701f3cb0 0x7f1270158730 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12701f6f90 0x7f12701f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:13.070 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 --2- 172.21.15.46:0/890738210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f125c005180 0x7f12700cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:13.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 >> 172.21.15.46:0/890738210 conn(0x7f1270070e70 msgr2=0x7f12701419d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:13.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 shutdown_connections 2023-12-16T21:16:13.071 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:13.070+0000 7f1277e5c640 1 -- 172.21.15.46:0/890738210 wait complete. 2023-12-16T21:16:13.080 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2023-12-16T21:16:11.992243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238972,"kb_used_data":6152,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735748,"statfs":{"total":773094113280,"available":772849405952,"internally_reserved":0,"allocated":6299648,"data_stored":3294488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17093,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-68,"num_write_kb":-41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001625"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":194,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-16T21:16:04.303970+0000","last_change":"2023-12-16T21:13:26.050043+0000","last_active":"2023-12-16T21:16:04.303970+0000","last_peered":"2023-12-16T21:16:04.303970+0000","last_clean":"2023-12-16T21:16:04.303970+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:16:04.303970+0000","last_undegraded":"2023-12-16T21:16:04.303970+0000","last_fullsized":"2023-12-16T21:16:04.303970+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:13:24.973043+0000","last_clean_scrub_stamp":"2023-12-16T21:13:24.973043+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T23:21:40.866468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1232896,"data_stored":854587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.65000000000000002,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.42699999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.628}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67800000000000005,"5min":1.159,"15min":1.159},"last":0.438}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69299999999999995,"5min":1.181,"15min":1.181},"last":0.54700000000000004}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.38,"5min":0.372,"15min":0.372},"max":{"1min":0.72999999999999998,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.624,"15min":0.624},"min":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.749,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61199999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67700000000000005,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.52100000000000002,"15min":0.51800000000000002},"min":{"1min":0.39400000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.495}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.53100000000000003,"15min":0.52900000000000003},"min":{"1min":0.40500000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56699999999999995,"15min":0.56499999999999995},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.752,"15min":0.752},"last":0.68000000000000005}]},{"osd":6,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.75600000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58699999999999997,"15min":0.58299999999999996},"min":{"1min":0.42099999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76800000000000002,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59999999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.40300000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53100000000000003,"15min":0.52200000000000002},"min":{"1min":0.377,"5min":0.373,"15min":0.373},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.66000000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48499999999999999,"15min":0.47199999999999998},"min":{"1min":0.38300000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":0.755,"15min":0.755},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53600000000000003,"15min":0.52800000000000002},"min":{"1min":0.36099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.63}]},{"osd":2,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.442,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53300000000000003,"15min":0.52300000000000002},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72899999999999998,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55200000000000005,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.40000000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81399999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.48399999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51800000000000002,"15min":0.50800000000000001},"min":{"1min":0.42299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.38700000000000001,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.505}]},{"osd":7,"last update":"Sat Dec 16 21:15:24 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.51500000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.872,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.57099999999999995,"15min":0.56000000000000005},"min":{"1min":0.42099999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51600000000000001,"15min":0.51900000000000002},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64600000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.47099999999999997,"15min":0.46200000000000002},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.51600000000000001}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51000000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.70899999999999996,"15min":0.73899999999999999},"min":{"1min":0.38600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.72199999999999998,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.57699999999999996}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.46700000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.68799999999999994,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53700000000000003,"15min":0.53900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61599999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.42599999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70699999999999996,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56299999999999994,"15min":0.56499999999999995},"min":{"1min":0.42499999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.69399999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.67500000000000004}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.51000000000000001,"15min":0.52100000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.75},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65800000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.86399999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54200000000000004,"15min":0.55000000000000004},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.56299999999999994},"min":{"1min":0.40500000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70299999999999996,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.82399999999999995}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.73299999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.44500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.752,"15min":0.752},"last":0.77500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.497}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.38},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.48399999999999999}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46300000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54500000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.28599999999999998}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.70999999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.82699999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.67700000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36099999999999999,"15min":0.36299999999999999},"min":{"1min":0.246,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.55000000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.309},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.41299999999999998,"15min":0.41799999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.68700000000000006,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.34399999999999997}]},{"osd":1,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.433,"15min":0.438},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.58499999999999996,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41799999999999998,"15min":0.42199999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.624,"5min":1.367,"15min":1.367},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.41399999999999998,"15min":0.42199999999999999},"min":{"1min":0.27100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.56399999999999995,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.46300000000000002,"15min":0.46899999999999997},"min":{"1min":0.32100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65700000000000003,"5min":1.46,"15min":1.46},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.41999999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.58799999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78800000000000003,"5min":0.874,"15min":0.874},"last":0.52400000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60699999999999998,"15min":0.61099999999999999},"min":{"1min":0.436,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81499999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61899999999999999,"15min":0.623},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76200000000000001,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50700000000000001}]},{"osd":6,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.62,"15min":0.63},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72399999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.621},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.625,"15min":0.63100000000000001},"min":{"1min":0.40999999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54100000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.625,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80100000000000005,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63200000000000001,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.28799999999999998}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.316},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38200000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.67300000000000004}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.623}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60499999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32384,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339456,"statfs":{"total":96636764160,"available":96603602944,"internally_reserved":0,"allocated":1232896,"data_stored":854587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.38}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.26000000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.40200000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.443},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.45700000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.47299999999999998}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.433},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48999999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.50600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:16:13.085 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-16T21:13:24.973043+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=21, tm_min=13, tm_sec=24, tm_wday=5, tm_yday=350, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=16, tm_hour=21, tm_min=16, tm_sec=8, tm_wday=5, tm_yday=350, tm_isdst=0) 2023-12-16T21:16:13.085 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-16T21:16:28.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:16:28.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:16:33.085 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-16T21:16:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.204+0000 7fccf0b90640 1 Processor -- start 2023-12-16T21:16:33.205 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.204+0000 7fccf0b90640 1 -- start start 2023-12-16T21:16:33.206 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccf0b90640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccec143490 0x7fccec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccf0b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec15e730 0x7fccec0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccf0b90640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fccec15a4e0 0x7fccec15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccf0b90640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccec15b520 con 0x7fccec143490 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccf0b90640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccec143870 con 0x7fccec15a4e0 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccf0b90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec15e2a0 con 0x7fccec15e730 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccea59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccec143490 0x7fccec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.207 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccea59c640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7fccec143490 0x7fccec0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57534/0 (socket says 172.21.15.46:57534) 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccea59c640 1 -- 172.21.15.46:0/2188034003 learned_addr learned my addr 172.21.15.46:0/2188034003 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fccead9d640 1 --2- 172.21.15.46:0/2188034003 >> v2:172.21.15.121:3300/0 conn(0x7fccec15a4e0 0x7fccec15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.205+0000 7fcce9d9b640 1 --2- 172.21.15.46:0/2188034003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec15e730 0x7fccec0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccea59c640 1 -- 172.21.15.46:0/2188034003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec15e730 msgr2=0x7fccec0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccea59c640 1 --2- 172.21.15.46:0/2188034003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec15e730 0x7fccec0b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccea59c640 1 -- 172.21.15.46:0/2188034003 >> v2:172.21.15.121:3300/0 conn(0x7fccec15a4e0 msgr2=0x7fccec15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:33.208 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccea59c640 1 --2- 172.21.15.46:0/2188034003 >> v2:172.21.15.121:3300/0 conn(0x7fccec15a4e0 0x7fccec15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccea59c640 1 -- 172.21.15.46:0/2188034003 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec15b240 con 0x7fccec143490 2023-12-16T21:16:33.209 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccead9d640 1 --2- 172.21.15.46:0/2188034003 >> v2:172.21.15.121:3300/0 conn(0x7fccec15a4e0 0x7fccec15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fccea59c640 1 --2- 172.21.15.46:0/2188034003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec143490 0x7fccec0b0cf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fccec15eb90 tx=0x7fcce402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50586c6c790c35e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.206+0000 7fcce959a640 1 -- 172.21.15.46:0/2188034003 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4031870 con 0x7fccec143490 2023-12-16T21:16:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fcce959a640 1 -- 172.21.15.46:0/2188034003 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce40319d0 con 0x7fccec143490 2023-12-16T21:16:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fcce959a640 1 -- 172.21.15.46:0/2188034003 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce4031d60 con 0x7fccec143490 2023-12-16T21:16:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 -- 172.21.15.46:0/2188034003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec143490 msgr2=0x7fccec0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.212 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 --2- 172.21.15.46:0/2188034003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec143490 0x7fccec0b0cf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fccec15eb90 tx=0x7fcce402b910 comp rx=0 tx=0).stop 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 -- 172.21.15.46:0/2188034003 shutdown_connections 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 --2- 172.21.15.46:0/2188034003 >> v2:172.21.15.121:3300/0 conn(0x7fccec15a4e0 0x7fccec15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 --2- 172.21.15.46:0/2188034003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec15e730 0x7fccec0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 --2- 172.21.15.46:0/2188034003 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec143490 0x7fccec0b0cf0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 -- 172.21.15.46:0/2188034003 >> 172.21.15.46:0/2188034003 conn(0x7fccec06a4d0 msgr2=0x7fccec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 -- 172.21.15.46:0/2188034003 shutdown_connections 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.207+0000 7fccf0b90640 1 -- 172.21.15.46:0/2188034003 wait complete. 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 Processor -- start 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 -- start start 2023-12-16T21:16:33.213 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec143490 0x7fccec1599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 0x7fccec13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccec15e730 0x7fccec13d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec13e960 con 0x7fccec15a4e0 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec159ef0 con 0x7fccec15e730 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.209+0000 7fccf0b90640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec13dd50 con 0x7fccec143490 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 0x7fccec13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 0x7fccec13cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57536/0 (socket says 172.21.15.46:57536) 2023-12-16T21:16:33.214 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 -- 172.21.15.46:0/1119410306 learned_addr learned my addr 172.21.15.46:0/1119410306 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccea59c640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec143490 0x7fccec1599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccead9d640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccec15e730 0x7fccec13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 -- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec143490 msgr2=0x7fccec1599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec143490 0x7fccec1599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 -- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccec15e730 msgr2=0x7fccec13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccec15e730 0x7fccec13d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 -- 172.21.15.46:0/1119410306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40094d0 con 0x7fccec15a4e0 2023-12-16T21:16:33.215 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fcce9d9b640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 0x7fccec13cfe0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcce000d5f0 tx=0x7fcce000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce00125f0 con 0x7fccec15a4e0 2023-12-16T21:16:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce0012750 con 0x7fccec15a4e0 2023-12-16T21:16:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec13e030 con 0x7fccec15a4e0 2023-12-16T21:16:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0012b00 con 0x7fccec15a4e0 2023-12-16T21:16:33.216 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.210+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec13e5f0 con 0x7fccec15a4e0 2023-12-16T21:16:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.211+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec13eb10 con 0x7fccec15a4e0 2023-12-16T21:16:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.212+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7fcce0012c60 con 0x7fccec15a4e0 2023-12-16T21:16:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.212+0000 7fccdaffd640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcccc06ae40 0x7fcccc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.213+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcce0089670 con 0x7fccec15a4e0 2023-12-16T21:16:33.217 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.213+0000 7fccea59c640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcccc06ae40 0x7fcccc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.213+0000 7fccea59c640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcccc06ae40 0x7fcccc06d300 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcce4036000 tx=0x7fcce403c040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.218 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.217+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7fccec13eb10 con 0x7fccec15a4e0 2023-12-16T21:16:33.391 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.390+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 --> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccec15aeb0 con 0x7fcccc06ae40 2023-12-16T21:16:33.393 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.393+0000 7fccdaffd640 1 -- 172.21.15.46:0/1119410306 <== mgr.4108 v2:172.21.15.121:6832/362353154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50881 (secure 0 0 0) 0x7fcce4031b40 con 0x7fcccc06ae40 2023-12-16T21:16:33.394 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:33.395 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2023-12-16T21:16:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcccc06ae40 msgr2=0x7fcccc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcccc06ae40 0x7fcccc06d300 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcce4036000 tx=0x7fcce403c040 comp rx=0 tx=0).stop 2023-12-16T21:16:33.396 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 msgr2=0x7fccec13cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 0x7fccec13cfe0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fcce000d5f0 tx=0x7fcce000dab0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 shutdown_connections 2023-12-16T21:16:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7fcccc06ae40 0x7fcccc06d300 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fccec15a4e0 0x7fccec13cfe0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.397 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccec15e730 0x7fccec13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 --2- 172.21.15.46:0/1119410306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccec143490 0x7fccec1599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.394+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 >> 172.21.15.46:0/1119410306 conn(0x7fccec06a4d0 msgr2=0x7fccec070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.395+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 shutdown_connections 2023-12-16T21:16:33.398 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.395+0000 7fccf0b90640 1 -- 172.21.15.46:0/1119410306 wait complete. 2023-12-16T21:16:33.406 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2023-12-16T21:16:31.996115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238964,"kb_used_data":6152,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735756,"statfs":{"total":773094113280,"available":772849414144,"internally_reserved":0,"allocated":6299648,"data_stored":3294488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12753,"internal_metadata":238276143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002475"},"pg_stats":[{"pgid":"1.0","version":"53'119","reported_seq":205,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-12-16T21:16:14.496504+0000","last_change":"2023-12-16T21:16:14.496549+0000","last_active":"2023-12-16T21:16:14.496504+0000","last_peered":"2023-12-16T21:16:14.496504+0000","last_clean":"2023-12-16T21:16:14.496504+0000","last_became_active":"2023-12-16T21:13:26.049056+0000","last_became_peered":"2023-12-16T21:13:26.049056+0000","last_unstale":"2023-12-16T21:16:14.496504+0000","last_undegraded":"2023-12-16T21:16:14.496504+0000","last_fullsized":"2023-12-16T21:16:14.496504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'119","last_scrub_stamp":"2023-12-16T21:16:14.496408+0000","last_deep_scrub":"53'119","last_deep_scrub_stamp":"2023-12-16T21:16:14.496408+0000","last_clean_scrub_stamp":"2023-12-16T21:16:14.496408+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T02:28:24.371952+0000","scrub_duration":0.0048972069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1232896,"data_stored":854587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54900000000000004,"15min":0.54200000000000004},"min":{"1min":0.47199999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69899999999999995,"5min":1.071,"15min":1.071},"last":0.67500000000000004}]},{"osd":1,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54600000000000004,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":1.159,"15min":1.159},"last":0.64400000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.374,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67400000000000004,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.42299999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.75,"5min":1.181,"15min":1.181},"last":0.75}]},{"osd":3,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59299999999999997,"15min":0.59999999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71699999999999997,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.80300000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.80300000000000005}]},{"osd":4,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51000000000000001,"15min":0.51800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.497},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.34300000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.51900000000000002}]},{"osd":5,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.621},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68899999999999995,"5min":0.752,"15min":0.752},"last":0.59499999999999997}]},{"osd":6,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60099999999999998,"15min":0.60399999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.72999999999999998,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70699999999999996,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.56899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.41899999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.78900000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54400000000000004,"15min":0.52600000000000002},"min":{"1min":0.39900000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.80700000000000005,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.77000000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.502,"15min":0.47799999999999998},"min":{"1min":0.45100000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73499999999999999,"5min":0.755,"15min":0.755},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.73399999999999999,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.495}]},{"osd":2,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53300000000000003,"15min":0.50900000000000001},"min":{"1min":0.46899999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72499999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.62},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54700000000000004,"15min":0.52800000000000002},"min":{"1min":0.41699999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.755,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.748}]},{"osd":3,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.442,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71599999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56299999999999994,"15min":0.55100000000000005},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66500000000000004}]},{"osd":4,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52700000000000002,"15min":0.51800000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73299999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.32800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79200000000000004,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.79200000000000004}]},{"osd":5,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.373,"5min":0.307,"15min":0.307},"max":{"1min":0.75800000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56200000000000006,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77200000000000002,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Sat Dec 16 21:16:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72599999999999998,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57799999999999996,"15min":0.56200000000000006},"min":{"1min":0.376,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78800000000000003,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.371,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.499,"15min":0.47099999999999997},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.81399999999999995}]},{"osd":1,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.52600000000000002,"15min":0.51200000000000001},"min":{"1min":0.42899999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.67800000000000005,"15min":0.72899999999999998},"min":{"1min":0.432,"5min":0.374,"15min":0.374},"max":{"1min":0.77700000000000002,"5min":5.3639999999999999,"15min":5.3639999999999999},"last":0.77700000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.52500000000000002},"min":{"1min":0.41099999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.55400000000000005,"15min":0.54500000000000004},"min":{"1min":0.498,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.83499999999999996}]},{"osd":3,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.49099999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.79800000000000004}]},{"osd":4,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.502,"15min":0.51800000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.495},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.54700000000000004},"min":{"1min":0.37,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.63},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55100000000000005,"15min":0.56100000000000005},"min":{"1min":0.35299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54300000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:16:27 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.57299999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.61399999999999999,"15min":0.629},"min":{"1min":0.442,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.68999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27648,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69799999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.318,"15min":0.318},"max":{"1min":0.79200000000000004,"5min":0.997,"15min":0.997},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.71499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.748,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57299999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.45700000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.54000000000000004,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71899999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.81299999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.504,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.38700000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65800000000000003,"5min":1.044,"15min":1.044},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.59399999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.34200000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77700000000000002,"5min":1.012,"15min":1.012},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68799999999999994}]}]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35299999999999998},"min":{"1min":0.23899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.61399999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38400000000000001,"15min":0.38600000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.54400000000000004,"5min":1.008,"15min":1.008},"last":0.375}]},{"osd":1,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40000000000000002,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.58599999999999997,"5min":1.329,"15min":1.329},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.0860000000000001,"5min":1.266,"15min":1.266},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.27100000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.72899999999999998,"5min":1.298,"15min":1.298},"last":0.39500000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.46800000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.48599999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.49399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98699999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.47999999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.51700000000000002}]},{"osd":6,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.51200000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.008,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.42399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Dec 16 21:15:26 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.627,"15min":0.623},"min":{"1min":0.438,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.497,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93700000000000006,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.499}]}]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31808,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35699999999999998,"15min":0.36199999999999999},"min":{"1min":0.26100000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.502,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.37},{"interface":"front","average":{"1min":0.379,"5min":0.40400000000000003,"15min":0.41499999999999998},"min":{"1min":0.26700000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.61799999999999999,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42199999999999999,"15min":0.434},"min":{"1min":0.26500000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.53600000000000003,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.433},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41399999999999998,"15min":0.42099999999999999},"min":{"1min":0.30399999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55400000000000005,"5min":1.367,"15min":1.367},"last":0.44500000000000001}]},{"osd":3,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.40600000000000003,"15min":0.41999999999999998},"min":{"1min":0.25900000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.54700000000000004,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.45000000000000001,"15min":0.46400000000000002},"min":{"1min":0.35699999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.54000000000000004,"5min":1.46,"15min":1.46},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59899999999999998,"15min":0.60899999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78600000000000003,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58699999999999997,"15min":0.59099999999999997},"min":{"1min":0.41999999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76600000000000001,"5min":0.874,"15min":0.874},"last":0.60499999999999998}]},{"osd":5,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.60999999999999999},"min":{"1min":0.46400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.44700000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.78000000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.748}]},{"osd":6,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61299999999999999,"15min":0.627},"min":{"1min":0.44400000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.80200000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.626,"15min":0.63100000000000001},"min":{"1min":0.47399999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82599999999999996,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.82599999999999996}]},{"osd":7,"last update":"Sat Dec 16 21:16:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61899999999999999,"15min":0.626},"min":{"1min":0.48699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81200000000000006,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.48699999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.82899999999999996,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.80300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31804,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":638976,"data_stored":264219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32700000000000001,"15min":0.32800000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.47999999999999998,"5min":0.504,"15min":0.504},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.36299999999999999,"15min":0.36099999999999999},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52900000000000003,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.32700000000000001}]},{"osd":2,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.33900000000000002,"15min":0.33500000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56299999999999994,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.41499999999999998}]},{"osd":3,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.28699999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54300000000000004,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.57199999999999995,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.57499999999999996,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.63400000000000001,"15min":0.628},"min":{"1min":0.47399999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.88600000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.61099999999999999}]},{"osd":5,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58699999999999997,"15min":0.58099999999999996},"min":{"1min":0.46300000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61699999999999999,"15min":0.61099999999999999},"min":{"1min":0.504,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86199999999999999,"5min":1.107,"15min":1.107},"last":0.86299999999999999}]},{"osd":6,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.627,"15min":0.624},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.90800000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.46100000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86099999999999999,"5min":1.153,"15min":1.153},"last":0.84099999999999997}]},{"osd":7,"last update":"Sat Dec 16 21:15:27 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63900000000000001,"15min":0.63500000000000001},"min":{"1min":0.47599999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.90600000000000003,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.629,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.81399999999999995,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.77700000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1232896,"data_stored":854587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34699999999999998,"15min":0.34899999999999998},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.245,"15min":0.245},"max":{"1min":0.39300000000000002,"5min":0.41999999999999998,"15min":0.41999999999999998},"last":0.36699999999999999}]},{"osd":2,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35199999999999998,"15min":0.35399999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.41499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.37,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.46400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"min":{"1min":0.29799999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.41299999999999998,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.40400000000000003,"15min":0.40799999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.71599999999999997,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56100000000000005}]},{"osd":5,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.59499999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.58099999999999996}]},{"osd":6,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.68400000000000005,"5min":0.752,"15min":0.752},"last":0.626},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.60899999999999999}]},{"osd":7,"last update":"Sat Dec 16 21:15:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.42599999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.61599999999999999,"15min":0.62},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:16:33.408 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-16T21:16:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 Processor -- start 2023-12-16T21:16:33.515 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 -- start start 2023-12-16T21:16:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5aec143490 0x7f5aec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aec15abf0 0x7f5aec0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec15e2a0 con 0x7f5aec15e730 2023-12-16T21:16:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec15b520 con 0x7f5aec15abf0 2023-12-16T21:16:33.518 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.514+0000 7f5af15dc640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec143870 con 0x7f5aec143490 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aec15abf0 0x7f5aec0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f5aec15abf0 0x7f5aec0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57540/0 (socket says 172.21.15.46:57540) 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 -- 172.21.15.46:0/873055232 learned_addr learned my addr 172.21.15.46:0/873055232 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeaffd640 1 --2- 172.21.15.46:0/873055232 >> v2:172.21.15.121:3300/0 conn(0x7f5aec143490 0x7f5aec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 -- 172.21.15.46:0/873055232 >> v2:172.21.15.121:3300/0 conn(0x7f5aec143490 msgr2=0x7f5aec0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 --2- 172.21.15.46:0/873055232 >> v2:172.21.15.121:3300/0 conn(0x7f5aec143490 0x7f5aec0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 -- 172.21.15.46:0/873055232 >> v1:172.21.15.121:6789/0 conn(0x7f5aec15e730 legacy=0x7f5aec15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeb7fe640 1 -- 172.21.15.46:0/873055232 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec15b240 con 0x7f5aec15abf0 2023-12-16T21:16:33.519 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.515+0000 7f5aeaffd640 1 --2- 172.21.15.46:0/873055232 >> v2:172.21.15.121:3300/0 conn(0x7f5aec143490 0x7f5aec0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5aeb7fe640 1 --2- 172.21.15.46:0/873055232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec15abf0 0x7f5aec0b1230 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06e620 tx=0x7f5adc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4921585db3dcb523 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5ae9ffb640 1 -- 172.21.15.46:0/873055232 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc02cb20 con 0x7f5aec15abf0 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5ae9ffb640 1 -- 172.21.15.46:0/873055232 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc02cc80 con 0x7f5aec15abf0 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5ae9ffb640 1 -- 172.21.15.46:0/873055232 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc0343f0 con 0x7f5aec15abf0 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5af15dc640 1 -- 172.21.15.46:0/873055232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec15abf0 msgr2=0x7f5aec0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5af15dc640 1 --2- 172.21.15.46:0/873055232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec15abf0 0x7f5aec0b1230 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f5aec06e620 tx=0x7f5adc02bf80 comp rx=0 tx=0).stop 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5af15dc640 1 -- 172.21.15.46:0/873055232 shutdown_connections 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5af15dc640 1 --2- 172.21.15.46:0/873055232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec15abf0 0x7f5aec0b1230 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.520 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5af15dc640 1 --2- 172.21.15.46:0/873055232 >> v2:172.21.15.121:3300/0 conn(0x7f5aec143490 0x7f5aec0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.516+0000 7f5af15dc640 1 -- 172.21.15.46:0/873055232 >> 172.21.15.46:0/873055232 conn(0x7f5aec06a4d0 msgr2=0x7f5aec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.517+0000 7f5af15dc640 1 -- 172.21.15.46:0/873055232 shutdown_connections 2023-12-16T21:16:33.521 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.517+0000 7f5af15dc640 1 -- 172.21.15.46:0/873055232 wait complete. 2023-12-16T21:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.518+0000 7f5af15dc640 1 Processor -- start 2023-12-16T21:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.518+0000 7f5af15dc640 1 -- start start 2023-12-16T21:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 0x7f5aec0ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5aec15abf0 0x7f5aec1e7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5aec15e730 0x7f5aec159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0ad8d0 con 0x7f5aec143490 2023-12-16T21:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0adbb0 con 0x7f5aec15e730 2023-12-16T21:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0ade90 con 0x7f5aec15abf0 2023-12-16T21:16:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 0x7f5aec0ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 0x7f5aec0ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57546/0 (socket says 172.21.15.46:57546) 2023-12-16T21:16:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 -- 172.21.15.46:0/4080407053 learned_addr learned my addr 172.21.15.46:0/4080407053 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeb7fe640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5aec15e730 0x7f5aec159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 -- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5aec15abf0 msgr2=0x7f5aec1e7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5aec15abf0 0x7f5aec1e7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 -- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5aec15e730 msgr2=0x7f5aec159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5aec15e730 0x7f5aec159a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 -- 172.21.15.46:0/4080407053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc009b30 con 0x7f5aec143490 2023-12-16T21:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5aeaffd640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 0x7f5aec0ad2f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400e730 tx=0x7f5ae400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.528 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4004230 con 0x7f5aec143490 2023-12-16T21:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae4004390 con 0x7f5aec143490 2023-12-16T21:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec0ae170 con 0x7f5aec143490 2023-12-16T21:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400c8f0 con 0x7f5aec143490 2023-12-16T21:16:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.519+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec159f50 con 0x7f5aec143490 2023-12-16T21:16:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.520+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec0ae360 con 0x7f5aec143490 2023-12-16T21:16:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.521+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f5ae400ca50 con 0x7f5aec143490 2023-12-16T21:16:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.521+0000 7f5ae37fe640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5acc06ae40 0x7f5acc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.521+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5ae4088f60 con 0x7f5aec143490 2023-12-16T21:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.522+0000 7f5aea7fc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5acc06ae40 0x7f5acc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.522+0000 7f5aea7fc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5acc06ae40 0x7f5acc06d300 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4000f60 tx=0x7f5ad40079b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.525+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f5ae40542f0 con 0x7f5aec143490 2023-12-16T21:16:33.703 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.702+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f5aec0b4380 con 0x7f5aec143490 2023-12-16T21:16:33.705 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.705+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) v1 ==== 123+0+0 (secure 0 0 0) 0x7f5ae4057910 con 0x7f5aec143490 2023-12-16T21:16:33.706 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.705+0000 7f5ae37fe640 1 -- 172.21.15.46:0/4080407053 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5ae4004500 con 0x7f5aec143490 2023-12-16T21:16:33.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5acc06ae40 msgr2=0x7f5acc06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.708 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5acc06ae40 0x7f5acc06d300 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4000f60 tx=0x7f5ad40079b0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 msgr2=0x7f5aec0ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 0x7f5aec0ad2f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400e730 tx=0x7f5ae400ebf0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 shutdown_connections 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f5acc06ae40 0x7f5acc06d300 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5aec15e730 0x7f5aec159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5aec15abf0 0x7f5aec1e7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 --2- 172.21.15.46:0/4080407053 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5aec143490 0x7f5aec0ad2f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.709 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 >> 172.21.15.46:0/4080407053 conn(0x7f5aec06a4d0 msgr2=0x7f5aec0b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:33.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 shutdown_connections 2023-12-16T21:16:33.710 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.707+0000 7f5af15dc640 1 -- 172.21.15.46:0/4080407053 wait complete. 2023-12-16T21:16:33.719 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-16T21:16:33.719 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:16:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.825+0000 7f69c929b640 1 Processor -- start 2023-12-16T21:16:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.825+0000 7f69c929b640 1 -- start start 2023-12-16T21:16:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c929b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c4146aa0 0x7f69c40b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c929b640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69c40b4ed0 0x7f69c40b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c929b640 1 -- --> v1:172.21.15.46:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69c40b1690 con 0x7f69c40b0cf0 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c929b640 1 -- --> v2:172.21.15.46:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69c4161bb0 con 0x7f69c40b4ed0 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c929b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c4160fe0 con 0x7f69c4146aa0 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c37fe640 1 --2- >> v2:172.21.15.46:3300/0 conn(0x7f69c40b4ed0 0x7f69c40b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69bbfff640 1 --1- >> v1:172.21.15.46:6789/0 conn(0x7f69c40b0cf0 0x7f69c40ac8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.46:6789/0 says I am v1:172.21.15.46:34620/0 (socket says 172.21.15.46:34620) 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69bbfff640 1 -- 172.21.15.46:0/2735174886 learned_addr learned my addr 172.21.15.46:0/2735174886 (peer_addr_for_me v1:172.21.15.46:0/0) 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c37fe640 1 --2- 172.21.15.46:0/2735174886 >> v2:172.21.15.46:3300/0 conn(0x7f69c40b4ed0 0x7f69c40b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57558/0 (socket says 172.21.15.46:57558) 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c2ffd640 1 --2- 172.21.15.46:0/2735174886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c4146aa0 0x7f69c40b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c37fe640 1 -- 172.21.15.46:0/2735174886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c4146aa0 msgr2=0x7f69c40b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-16T21:16:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c37fe640 1 --2- 172.21.15.46:0/2735174886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c4146aa0 0x7f69c40b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c37fe640 1 -- 172.21.15.46:0/2735174886 >> v1:172.21.15.46:6789/0 conn(0x7f69c40b0cf0 legacy=0x7f69c40ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c37fe640 1 -- 172.21.15.46:0/2735174886 --> v2:172.21.15.46:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c4146e80 con 0x7f69c40b4ed0 2023-12-16T21:16:33.829 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.826+0000 7f69c27fc640 1 -- 172.21.15.46:0/2735174886 <== mon.0 v1:172.21.15.46:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008177581 0 0) 0x7f69c40b1690 con 0x7f69c40b0cf0 2023-12-16T21:16:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.827+0000 7f69c37fe640 1 --2- 172.21.15.46:0/2735174886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c40b4ed0 0x7f69c40b1150 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009d90 tx=0x7f69bc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2203509a87350259 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.827+0000 7f69c27fc640 1 -- 172.21.15.46:0/2735174886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc02cbc0 con 0x7f69c40b4ed0 2023-12-16T21:16:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.827+0000 7f69c27fc640 1 -- 172.21.15.46:0/2735174886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f69bc02cd20 con 0x7f69c40b4ed0 2023-12-16T21:16:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.827+0000 7f69c27fc640 1 -- 172.21.15.46:0/2735174886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc034680 con 0x7f69c40b4ed0 2023-12-16T21:16:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 -- 172.21.15.46:0/2735174886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c40b4ed0 msgr2=0x7f69c40b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.832 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 --2- 172.21.15.46:0/2735174886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c40b4ed0 0x7f69c40b1150 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f69bc009d90 tx=0x7f69bc02be10 comp rx=0 tx=0).stop 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 -- 172.21.15.46:0/2735174886 shutdown_connections 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 --2- 172.21.15.46:0/2735174886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c40b4ed0 0x7f69c40b1150 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 --2- 172.21.15.46:0/2735174886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c4146aa0 0x7f69c40b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 -- 172.21.15.46:0/2735174886 >> 172.21.15.46:0/2735174886 conn(0x7f69c406a4d0 msgr2=0x7f69c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 -- 172.21.15.46:0/2735174886 shutdown_connections 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.828+0000 7f69c929b640 1 -- 172.21.15.46:0/2735174886 wait complete. 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.829+0000 7f69c929b640 1 Processor -- start 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.829+0000 7f69c929b640 1 -- start start 2023-12-16T21:16:33.833 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c929b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69c40b0cf0 0x7f69c415f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c929b640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c40b4ed0 0x7f69c4157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c929b640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 0x7f69c4139430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c929b640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41593a0 con 0x7f69c4146aa0 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c929b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c4158300 con 0x7f69c40b0cf0 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c929b640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41585e0 con 0x7f69c40b4ed0 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 0x7f69c4139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 0x7f69c4139430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57574/0 (socket says 172.21.15.46:57574) 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 -- 172.21.15.46:0/992483937 learned_addr learned my addr 172.21.15.46:0/992483937 (peer_addr_for_me v2:172.21.15.46:0/0) 2023-12-16T21:16:33.834 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69bbfff640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c40b4ed0 0x7f69c4157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c2ffd640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69c40b0cf0 0x7f69c415f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 -- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c40b4ed0 msgr2=0x7f69c4157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c40b4ed0 0x7f69c4157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 -- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69c40b0cf0 msgr2=0x7f69c415f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69c40b0cf0 0x7f69c415f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.830+0000 7f69c37fe640 1 -- 172.21.15.46:0/992483937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc009a70 con 0x7f69c4146aa0 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.831+0000 7f69c37fe640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 0x7f69c4139430 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f69bc037040 tx=0x7f69bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.835 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.831+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc040650 con 0x7f69c4146aa0 2023-12-16T21:16:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.831+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f69bc02ce90 con 0x7f69c4146aa0 2023-12-16T21:16:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.831+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c41588c0 con 0x7f69c4146aa0 2023-12-16T21:16:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.831+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc040a10 con 0x7f69c4146aa0 2023-12-16T21:16:33.836 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.831+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c4158e80 con 0x7f69c4146aa0 2023-12-16T21:16:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.832+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c4159070 con 0x7f69c4146aa0 2023-12-16T21:16:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.832+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 89233+0+0 (secure 0 0 0) 0x7f69bc040b70 con 0x7f69c4146aa0 2023-12-16T21:16:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.833+0000 7f69bb7fe640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f69a406ae40 0x7f69a406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:16:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.833+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f69bc0ac000 con 0x7f69c4146aa0 2023-12-16T21:16:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.833+0000 7f69c2ffd640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f69a406ae40 0x7f69a406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:16:33.837 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.833+0000 7f69c2ffd640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f69a406ae40 0x7f69a406d300 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f69b40079a0 tx=0x7f69b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:16:33.838 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:33.837+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179842 (secure 0 0 0) 0x7f69bc0773d0 con 0x7f69c4146aa0 2023-12-16T21:16:34.050 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.050+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69c4069400 con 0x7f69c4146aa0 2023-12-16T21:16:34.051 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.050+0000 7f69bb7fe640 1 -- 172.21.15.46:0/992483937 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f69bc044e90 con 0x7f69c4146aa0 2023-12-16T21:16:34.052 INFO:teuthology.orchestra.run.smithi046.stdout: 2023-12-16T21:16:34.052 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5225,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6834/2022944732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":2022944732},{"type":"v1","addr":"172.21.15.46:6834","nonce":2022944732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2140776813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2140776813},{"type":"v1","addr":"172.21.15.121:6836","nonce":2140776813}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17},{"gid":5938,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/4235349931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":4235349931},{"type":"v1","addr":"172.21.15.121:6837","nonce":4235349931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5940,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/598614811","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6833","nonce":598614811},{"type":"v1","addr":"172.21.15.46:6835","nonce":598614811}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2023-12-16T21:16:34.052 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 22 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f69a406ae40 msgr2=0x7f69a406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f69a406ae40 0x7f69a406d300 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f69b40079a0 tx=0x7f69b4008040 comp rx=0 tx=0).stop 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 msgr2=0x7f69c4139430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 0x7f69c4139430 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f69bc037040 tx=0x7f69bc004a40 comp rx=0 tx=0).stop 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 shutdown_connections 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:6832/362353154,v1:172.21.15.121:6833/362353154] conn(0x7f69a406ae40 0x7f69a406d300 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:34.054 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f69c4146aa0 0x7f69c4139430 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f69c40b4ed0 0x7f69c4157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 --2- 172.21.15.46:0/992483937 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69c40b0cf0 0x7f69c415f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:16:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 >> 172.21.15.46:0/992483937 conn(0x7f69c406a4d0 msgr2=0x7f69c40b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:16:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 shutdown_connections 2023-12-16T21:16:34.055 INFO:teuthology.orchestra.run.smithi046.stderr:2023-12-16T21:16:34.052+0000 7f69c929b640 1 -- 172.21.15.46:0/992483937 wait complete. 2023-12-16T21:16:34.066 DEBUG:tasks.cephfs.filesystem:already dead... 2023-12-16T21:16:34.066 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-16T21:16:34.066 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-16T21:16:34.066 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.075 INFO:tasks.ceph.mds.a:Stopped 2023-12-16T21:16:34.076 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-16T21:16:34.076 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.125 INFO:tasks.ceph.mds.c:Stopped 2023-12-16T21:16:34.125 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-16T21:16:34.125 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.138 INFO:tasks.ceph.mds.b:Stopped 2023-12-16T21:16:34.138 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-16T21:16:34.138 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.189 INFO:tasks.ceph.mds.d:Stopped 2023-12-16T21:16:34.189 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-16T21:16:34.189 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-16T21:16:34.190 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.226 INFO:tasks.ceph.osd.0:Stopped 2023-12-16T21:16:34.226 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-16T21:16:34.226 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.291 INFO:tasks.ceph.osd.1:Stopped 2023-12-16T21:16:34.291 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-16T21:16:34.292 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.359 INFO:tasks.ceph.osd.2:Stopped 2023-12-16T21:16:34.359 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-16T21:16:34.359 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.427 INFO:tasks.ceph.osd.3:Stopped 2023-12-16T21:16:34.427 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-16T21:16:34.428 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.462 INFO:tasks.ceph.osd.4:Stopped 2023-12-16T21:16:34.463 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-16T21:16:34.463 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.531 INFO:tasks.ceph.osd.5:Stopped 2023-12-16T21:16:34.531 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-16T21:16:34.532 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.601 INFO:tasks.ceph.osd.6:Stopped 2023-12-16T21:16:34.601 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-16T21:16:34.601 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.670 INFO:tasks.ceph.osd.7:Stopped 2023-12-16T21:16:34.670 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-16T21:16:34.671 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-16T21:16:34.671 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.696 INFO:tasks.ceph.mgr.y:Stopped 2023-12-16T21:16:34.696 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-16T21:16:34.696 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.740 INFO:tasks.ceph.mgr.x:Stopped 2023-12-16T21:16:34.740 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-16T21:16:34.741 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-16T21:16:34.741 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.752 INFO:tasks.ceph.mon.a:Stopped 2023-12-16T21:16:34.752 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-16T21:16:34.752 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.792 INFO:tasks.ceph.mon.b:Stopped 2023-12-16T21:16:34.793 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-16T21:16:34.793 INFO:teuthology.orchestra.run:waiting for 300 2023-12-16T21:16:34.845 INFO:tasks.ceph.mon.c:Stopped 2023-12-16T21:16:34.845 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-16T21:16:34.845 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-16T21:16:34.881 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:16:34.882 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-16T21:16:35.109 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:16:35.109 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-16T21:16:35.233 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:16:35.234 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-16T21:16:35.299 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:16:35.300 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-16T21:16:35.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:16:35.349 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-16T21:16:35.527 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:16:35.527 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-16T21:16:35.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:16:35.606 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-16T21:16:35.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:16:35.664 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-16T21:16:35.729 INFO:tasks.ceph:Archiving mon data... 2023-12-16T21:16:35.730 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/data/mon.a.tgz 2023-12-16T21:16:35.730 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2023-12-16T21:16:35.747 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.2OYKIqBpAr 2023-12-16T21:16:35.747 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2OYKIqBpAr 2023-12-16T21:16:35.911 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.2OYKIqBpAr 2023-12-16T21:16:36.000 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.2OYKIqBpAr is 344KB 2023-12-16T21:16:36.096 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.2OYKIqBpAr 2023-12-16T21:16:36.111 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/data/mon.b.tgz 2023-12-16T21:16:36.112 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-12-16T21:16:36.129 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.aiLYXmPlCh 2023-12-16T21:16:36.130 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.aiLYXmPlCh 2023-12-16T21:16:36.301 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.aiLYXmPlCh 2023-12-16T21:16:36.387 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.aiLYXmPlCh is 454KB 2023-12-16T21:16:36.484 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.aiLYXmPlCh 2023-12-16T21:16:36.499 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/data/mon.c.tgz 2023-12-16T21:16:36.499 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-12-16T21:16:36.553 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.t5LewejK8c 2023-12-16T21:16:36.553 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.t5LewejK8c 2023-12-16T21:16:36.703 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.t5LewejK8c 2023-12-16T21:16:36.788 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.t5LewejK8c is 454KB 2023-12-16T21:16:36.886 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.t5LewejK8c 2023-12-16T21:16:36.901 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-16T21:16:36.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-16T21:16:36.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-16T21:16:36.995 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-16T21:16:37.022 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-16T21:16:37.022 INFO:tasks.ceph:Archiving crash dumps... 2023-12-16T21:16:37.024 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/remote/smithi046/crash 2023-12-16T21:16:37.025 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-16T21:16:37.069 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/remote/smithi121/crash 2023-12-16T21:16:37.070 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-16T21:16:37.096 INFO:tasks.ceph:Shutting down logrotate 2023-12-16T21:16:37.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:16:37.114 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-16T21:16:37.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-16T21:16:37.215 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-16T21:16:37.249 INFO:tasks.ceph:Compressing logs... 2023-12-16T21:16:37.250 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T21:16:37.258 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T21:16:37.349 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-16T21:16:37.349 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89152.log 2023-12-16T21:16:37.349 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.3.95689.log 2023-12-16T21:16:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.89152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89200.log 2023-12-16T21:16:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.3.95689.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89152.log.gz 2023-12-16T21:16:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-16T21:16:37.350 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.89200.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-12-16T21:16:37.350 INFO:teuthology.orchestra.run.smithi121.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.89200.log.gz 2023-12-16T21:16:37.351 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: 90.4% -- replaced with /var/log/ceph/ceph-client.3.95689.log.gz 2023-12-16T21:16:37.351 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89240.log 2023-12-16T21:16:37.351 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88951.log 2023-12-16T21:16:37.351 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mds.b.log: /var/log/ceph/ceph-client.admin.89240.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-16T21:16:37.351 INFO:teuthology.orchestra.run.smithi121.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.89240.log.gz 2023-12-16T21:16:37.351 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.88951.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-16T21:16:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-16T21:16:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.4.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.88951.log.gz 2023-12-16T21:16:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89031.log 2023-12-16T21:16:37.352 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.95237.log 2023-12-16T21:16:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.89031.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-16T21:16:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-16T21:16:37.353 INFO:teuthology.orchestra.run.smithi121.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.89031.log.gz 2023-12-16T21:16:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.1.95237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89071.log 2023-12-16T21:16:37.353 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-16T21:16:37.354 INFO:teuthology.orchestra.run.smithi121.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.1.95237.log.gz/var/log/ceph/ceph-client.admin.89071.log: 2023-12-16T21:16:37.354 INFO:teuthology.orchestra.run.smithi121.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-16T21:16:37.354 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89071.log.gz/var/log/ceph/ceph.log: 2023-12-16T21:16:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89111.log 2023-12-16T21:16:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.b.log: 92.2% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-16T21:16:37.354 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-12-16T21:16:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.89111.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89111.log.gz 2023-12-16T21:16:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88991.log 2023-12-16T21:16:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-16T21:16:37.355 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.88991.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88991.log.gz 2023-12-16T21:16:37.355 INFO:teuthology.orchestra.run.smithi121.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-12-16T21:16:37.357 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-12-16T21:16:37.377 INFO:teuthology.orchestra.run.smithi121.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-16T21:16:37.459 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104764.log 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103124.log 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108076.log 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109794.log 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104764.log.gz 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108076.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103124.log.gz 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112624.log 2023-12-16T21:16:37.460 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109794.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108076.log.gz 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113156.log 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.109794.log.gz 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104722.log 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109111.log 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112624.log.gz 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101606.log 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.113156.log.gz 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104722.log.gz 2023-12-16T21:16:37.461 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109111.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106012.log 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109111.log.gz 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101606.log.gz 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95990.log 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96508.log 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110266.log 2023-12-16T21:16:37.462 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106012.log.gz 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96508.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95990.log.gz 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110698.log 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96508.log.gz 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95906.log 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97376.log 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110266.log.gz 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95906.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110698.log.gz 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108645.log 2023-12-16T21:16:37.463 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95906.log.gz 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112078.log 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109878.log 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97376.log.gz 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112078.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108645.log.gz 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107094.log 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112078.log.gz 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108244.log 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.105369.log 2023-12-16T21:16:37.464 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109878.log.gz 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108244.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107094.log.gz 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96284.log 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108244.log.gz 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.105369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108202.log 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112582.log 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108202.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96284.log.gz 2023-12-16T21:16:37.465 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102668.log 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108202.log.gz 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95948.log 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112582.log.gz 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106826.log 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.0.105369.log.gz 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102668.log: /var/log/ceph/ceph-client.admin.95948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109027.log 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102668.log.gz 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95948.log.gz 2023-12-16T21:16:37.466 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98006.log 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103294.log 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106826.log.gz 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109027.log.gz 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100372.log 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101726.log 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100372.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.98006.log.gz 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.103294.log.gz 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95274.log 2023-12-16T21:16:37.467 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100372.log.gz 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106968.log 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101726.log.gz 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111866.log 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99159.log 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95274.log.gz 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111866.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106968.log.gz 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101480.log 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111866.log.gz 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113072.log 2023-12-16T21:16:37.468 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102272.log 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99159.log.gz 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113072.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101480.log.gz 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.83878.log 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113072.log.gz 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105970.log 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.83878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95440.log 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102272.log.gz 2023-12-16T21:16:37.469 INFO:teuthology.orchestra.run.smithi046.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.83878.log.gz 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107992.log 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97838.log 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105970.log.gz 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95440.log.gz 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107992.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107992.log.gz 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96158.log 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105618.log 2023-12-16T21:16:37.470 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107950.log 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.97838.log.gz 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105618.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96158.log.gz 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112162.log 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105618.log.gz 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104378.log 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112162.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.107950.log.gz 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106054.log 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104378.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112162.log.gz 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99941.log 2023-12-16T21:16:37.471 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106054.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104378.log.gz 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97502.log 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.106054.log.gz 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97670.log 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97502.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99941.log.gz 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102748.log 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97502.log.gz 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101296.log 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97670.log.gz 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104496.log 2023-12-16T21:16:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101296.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.102748.log.gz -5 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.112498.log 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101296.log.gz 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109153.log 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104496.log.gz/var/log/ceph/ceph-client.admin.112498.log: 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103876.log 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112498.log.gz 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99899.log 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102566.log 2023-12-16T21:16:37.473 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109153.log.gz 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99899.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103876.log.gz 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113114.log 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99899.log.gz 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110756.log 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98647.log 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102566.log.gz 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110756.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113114.log.gz 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95079.log 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110756.log.gz 2023-12-16T21:16:37.474 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112988.log 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108519.log 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98647.log.gz 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112988.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95079.log.gz 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105032.log 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112988.log.gz 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100844.log 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100330.log 2023-12-16T21:16:37.475 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108519.log.gz 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100844.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.105032.log.gz 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100640.log 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.100844.log.gz 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99647.log 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100330.log: /var/log/ceph/ceph-client.admin.100640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104906.log 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100330.log.gz 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99647.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100640.log.gz 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.99019.log 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99647.log.gz 2023-12-16T21:16:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108943.log 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104906.log: /var/log/ceph/ceph-client.2.99019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105786.log 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.104906.log.gz 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105702.log 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108943.log.gz 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111108.log 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.2.99019.log.gz 2023-12-16T21:16:37.477 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105786.log.gz 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110104.log 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97796.log 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105702.log.gz 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110104.log: gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.107511.log -- replaced with /var/log/ceph/ceph-client.admin.111108.log.gz 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102314.log 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110104.log.gz 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107511.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97796.log.gz 2023-12-16T21:16:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101012.log 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107511.log.gz 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113534.log 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111908.log 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102314.log.gz 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2%/var/log/ceph/ceph-client.admin.113534.log: -- replaced with /var/log/ceph/ceph-client.admin.101012.log.gz 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99311.log 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113534.log.gz 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96504.log 2023-12-16T21:16:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99395.log 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111908.log.gz 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96504.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99311.log.gz 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100682.log 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96504.log.gz 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.99395.log: -- /var/log/ceph/ceph-client.admin.112456.log 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96507.log 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99395.log.gz 2023-12-16T21:16:37.480 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112456.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100682.log.gz 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96506.log 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112456.log.gz 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111782.log 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96507.log.gz 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109920.log 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96506.log: /var/log/ceph/ceph-client.admin.111782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100414.log 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96506.log.gz 2023-12-16T21:16:37.481 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111782.log.gz 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100970.log 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109920.log.gz 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99857.log 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108775.log 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100414.log.gz 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100970.log.gz 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113366.log 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.99857.log.gz 2023-12-16T21:16:37.482 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99521.log 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97922.log 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108775.log.gz 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113366.log.gz 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101338.log 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.99521.log.gz 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106926.log 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97922.log: /var/log/ceph/ceph-client.admin.101338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97586.log 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97922.log.gz 2023-12-16T21:16:37.483 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101338.log.gz 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95524.log 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106926.log.gz 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.97586.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.113660.log 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96498.log 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.97586.log.gz 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95524.log.gz 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107908.log 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113660.log.gz 2023-12-16T21:16:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100109.log 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107865.log 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96498.log.gz 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107908.log.gz 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97179.log 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100109.log.gz 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113408.log 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107865.log: /var/log/ceph/ceph-client.admin.97179.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-16T21:16:37.485 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107865.log.gz 2023-12-16T21:16:37.486 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97179.log.gz 2023-12-16T21:16:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101438.log 2023-12-16T21:16:37.486 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113408.log.gz 2023-12-16T21:16:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104420.log 2023-12-16T21:16:37.486 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109736.log 2023-12-16T21:16:37.487 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101438.log.gz 2023-12-16T21:16:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108603.log 2023-12-16T21:16:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96835.log 2023-12-16T21:16:37.487 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104420.log.gz 2023-12-16T21:16:37.487 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109736.log: /var/log/ceph/ceph-client.admin.108603.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109736.log.gz 2023-12-16T21:16:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107763.log 2023-12-16T21:16:37.488 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-16T21:16:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97418.log 2023-12-16T21:16:37.488 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108603.log.gz 2023-12-16T21:16:37.488 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96835.log: /var/log/ceph/ceph-client.admin.107763.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96835.log.gz 2023-12-16T21:16:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100760.log 2023-12-16T21:16:37.489 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107763.log.gz 2023-12-16T21:16:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111661.log 2023-12-16T21:16:37.489 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106742.log 2023-12-16T21:16:37.490 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97418.log.gz 2023-12-16T21:16:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111661.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100760.log.gz 2023-12-16T21:16:37.490 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98108.log 2023-12-16T21:16:37.490 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111661.log.gz 2023-12-16T21:16:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104990.log 2023-12-16T21:16:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.98108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.111487.log 2023-12-16T21:16:37.491 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106742.log.gz 2023-12-16T21:16:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104990.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98108.log.gz 2023-12-16T21:16:37.491 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101380.log 2023-12-16T21:16:37.492 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.104990.log.gz 2023-12-16T21:16:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.111487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110924.log 2023-12-16T21:16:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107136.log 2023-12-16T21:16:37.492 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110924.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101380.log.gz 2023-12-16T21:16:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97176.log 2023-12-16T21:16:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107136.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110924.log.gz 2023-12-16T21:16:37.493 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103810.log 2023-12-16T21:16:37.493 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.111487.log.gz 2023-12-16T21:16:37.493 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.107136.log.gz 2023-12-16T21:16:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110982.log 2023-12-16T21:16:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95696.log 2023-12-16T21:16:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103810.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97176.log.gz 2023-12-16T21:16:37.494 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103166.log 2023-12-16T21:16:37.495 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110982.log.gz 2023-12-16T21:16:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101978.log 2023-12-16T21:16:37.495 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.103810.log.gz 2023-12-16T21:16:37.495 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113576.log 2023-12-16T21:16:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103166.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95696.log.gz 2023-12-16T21:16:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108118.log 2023-12-16T21:16:37.496 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103166.log.gz 2023-12-16T21:16:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113576.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101978.log.gz 2023-12-16T21:16:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112204.log 2023-12-16T21:16:37.496 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108118.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113576.log.gz 2023-12-16T21:16:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105660.log 2023-12-16T21:16:37.497 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.108118.log.gz 2023-12-16T21:16:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95864.log 2023-12-16T21:16:37.497 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109303.log 2023-12-16T21:16:37.498 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.112204.log.gz 2023-12-16T21:16:37.498 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95178.log 2023-12-16T21:16:37.498 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105660.log.gz 2023-12-16T21:16:37.498 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95864.log.gz 2023-12-16T21:16:37.498 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96840.log 2023-12-16T21:16:37.499 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109303.log.gz 2023-12-16T21:16:37.499 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99731.log 2023-12-16T21:16:37.499 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96840.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95178.log.gz 2023-12-16T21:16:37.499 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107010.log 2023-12-16T21:16:37.500 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96840.log.gz 2023-12-16T21:16:37.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104822.log 2023-12-16T21:16:37.500 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99731.log.gz 2023-12-16T21:16:37.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102440.log 2023-12-16T21:16:37.500 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107010.log.gz 2023-12-16T21:16:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102482.log 2023-12-16T21:16:37.501 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104822.log.gz 2023-12-16T21:16:37.501 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.110525.log 2023-12-16T21:16:37.501 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102440.log.gz 2023-12-16T21:16:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112540.log 2023-12-16T21:16:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.110525.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.102482.log.gz -- 2023-12-16T21:16:37.502 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.113720.log 2023-12-16T21:16:37.502 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99773.log 2023-12-16T21:16:37.503 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112540.log.gz 2023-12-16T21:16:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107427.log 2023-12-16T21:16:37.503 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113720.log.gz 2023-12-16T21:16:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105744.log 2023-12-16T21:16:37.503 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99773.log: /var/log/ceph/ceph-client.admin.107427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100514.log 2023-12-16T21:16:37.504 INFO:teuthology.orchestra.run.smithi046.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.110525.log.gz 2023-12-16T21:16:37.504 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99773.log.gz 2023-12-16T21:16:37.504 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95396.log 2023-12-16T21:16:37.504 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.107427.log.gz 2023-12-16T21:16:37.505 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100514.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105744.log.gzgzip 2023-12-16T21:16:37.505 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.95482.log 2023-12-16T21:16:37.505 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95396.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100514.log.gz 2023-12-16T21:16:37.505 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102626.log 2023-12-16T21:16:37.506 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95482.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95396.log.gz 2023-12-16T21:16:37.506 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110840.log 2023-12-16T21:16:37.506 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95482.log.gz 2023-12-16T21:16:37.506 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102626.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-16T21:16:37.506 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102626.log.gz 2023-12-16T21:16:37.507 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.110840.log: /var/log/ceph/ceph-client.admin.100025.log 2023-12-16T21:16:37.507 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96242.log 2023-12-16T21:16:37.507 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110840.log.gz 2023-12-16T21:16:37.507 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103082.log 2023-12-16T21:16:37.508 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100025.log.gz 2023-12-16T21:16:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101852.log 2023-12-16T21:16:37.508 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96242.log.gz 2023-12-16T21:16:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97754.log 2023-12-16T21:16:37.508 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103082.log: /var/log/ceph/ceph-client.admin.101852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96200.log 2023-12-16T21:16:37.509 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103082.log.gz 2023-12-16T21:16:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97754.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101852.log.gz 2023-12-16T21:16:37.509 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102998.log 2023-12-16T21:16:37.509 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97754.log.gz 2023-12-16T21:16:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103600.log 2023-12-16T21:16:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112246.log 2023-12-16T21:16:37.510 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96200.log.gz 2023-12-16T21:16:37.510 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102998.log: /var/log/ceph/ceph-client.admin.103600.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.102998.log.gz 2023-12-16T21:16:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101810.log 2023-12-16T21:16:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102398.log 2023-12-16T21:16:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112246.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103600.log.gz 2023-12-16T21:16:37.511 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110188.log 2023-12-16T21:16:37.512 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112246.log.gz 2023-12-16T21:16:37.512 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101810.log.gz 2023-12-16T21:16:37.512 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111740.log 2023-12-16T21:16:37.512 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102398.log.gz 2023-12-16T21:16:37.512 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95738.log 2023-12-16T21:16:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99201.log 2023-12-16T21:16:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111740.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.110188.log.gz 2023-12-16T21:16:37.513 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107385.log 2023-12-16T21:16:37.513 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111740.log.gz 2023-12-16T21:16:37.514 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95738.log.gz 2023-12-16T21:16:37.514 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109962.log 2023-12-16T21:16:37.514 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99983.log 2023-12-16T21:16:37.514 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99201.log.gz 2023-12-16T21:16:37.515 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7%/var/log/ceph/ceph-client.admin.109962.log: -- replaced with /var/log/ceph/ceph-client.admin.107385.log.gz 2023-12-16T21:16:37.515 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96501.log 2023-12-16T21:16:37.515 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109962.log.gz 2023-12-16T21:16:37.515 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101212.log 2023-12-16T21:16:37.516 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.99983.log.gzgzip 2023-12-16T21:16:37.516 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.103726.log 2023-12-16T21:16:37.516 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96501.log: /var/log/ceph/ceph-client.admin.101212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.98530.log 2023-12-16T21:16:37.516 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96501.log.gz 2023-12-16T21:16:37.517 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101212.log.gz 2023-12-16T21:16:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103430.log 2023-12-16T21:16:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103726.log: /var/log/ceph/ceph-client.0.98530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101768.log 2023-12-16T21:16:37.517 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103726.log.gz 2023-12-16T21:16:37.517 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108561.log 2023-12-16T21:16:37.517 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103430.log.gz 2023-12-16T21:16:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103684.log 2023-12-16T21:16:37.518 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.0.98530.log.gz 2023-12-16T21:16:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101936.log 2023-12-16T21:16:37.518 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101768.log.gz 2023-12-16T21:16:37.518 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108561.log: /var/log/ceph/ceph-client.admin.103684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100928.log 2023-12-16T21:16:37.518 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108561.log.gz 2023-12-16T21:16:37.519 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101936.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103684.log.gz 2023-12-16T21:16:37.519 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106784.log 2023-12-16T21:16:37.519 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101936.log.gz 2023-12-16T21:16:37.519 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108817.log 2023-12-16T21:16:37.519 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100928.log: /var/log/ceph/ceph-client.admin.106784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100193.log 2023-12-16T21:16:37.520 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100928.log.gz 2023-12-16T21:16:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108817.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.106784.log.gz 2023-12-16T21:16:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96843.log 2023-12-16T21:16:37.520 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108817.log.gz 2023-12-16T21:16:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109195.log 2023-12-16T21:16:37.520 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100193.log: /var/log/ceph/ceph-client.admin.96843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100802.log 2023-12-16T21:16:37.521 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100193.log.gz 2023-12-16T21:16:37.521 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89949.log 2023-12-16T21:16:37.521 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96843.log.gz 2023-12-16T21:16:37.521 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109195.log.gz 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95652.log 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.100802.log.gz 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.89949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111824.log 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102870.log 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113030.log 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95652.log.gz 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.89949.log.gz 88.8% 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111824.log.gz 2023-12-16T21:16:37.522 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108419.log 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.102870.log.gz 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.96116.log 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113030.log: /var/log/ceph/ceph-client.admin.108419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112820.log 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113030.log.gz 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96116.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108419.log.gz 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103250.log 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96116.log.gz 2023-12-16T21:16:37.523 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96838.log 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103558.log 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103250.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112820.log.gz 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96838.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103250.log.gz 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113324.log 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99815.log 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103558.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96838.log.gz 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113324.log: gzip -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.103558.log.gz -- 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.107343.log 2023-12-16T21:16:37.524 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96844.log 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99815.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.113324.log.gz 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107637.log 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99815.log.gz 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103768.log 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96844.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.107343.log.gz 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107637.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96844.log.gz 2023-12-16T21:16:37.525 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103040.log 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7%/var/log/ceph/ceph-client.admin.103768.log: -- replaced with /var/log/ceph/ceph-client.admin.107637.log.gz 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102062.log 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103768.log.gz 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.106451.log 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.103040.log.gz 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.102062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97040.log 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102062.log.gz 2023-12-16T21:16:37.526 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.106451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110146.log 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99479.log 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97040.log: /var/log/ceph/ceph-client.admin.110146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100288.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.97040.log.gz 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110146.log.gz 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108901.log 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99479.log.gz 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106192.log 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100288.log: /var/log/ceph/ceph-client.admin.108901.log: gzip -5 91.4% -- replaced with /var/log/ceph/ceph-client.0.106451.log.gz 2023-12-16T21:16:37.527 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100288.log.gz 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.113240.log 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106192.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108901.log.gz 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112904.log 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106192.log.gz 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102230.log 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113240.log: /var/log/ceph/ceph-client.admin.112904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110020.log 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113240.log.gz 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112904.log.gz 2023-12-16T21:16:37.528 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105576.log 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104336.log 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110020.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102230.log.gz 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102146.log 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110020.log.gz 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104336.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105576.log.gz 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90322.log 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.104336.log.gz 2023-12-16T21:16:37.529 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113198.log 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102146.log: /var/log/ceph/ceph-client.admin.90322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112372.log 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.102146.log.gz 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113198.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.90322.log.gz 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107805.log 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.113198.log.gz 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102104.log 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96837.log 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107805.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112372.log.gz 2023-12-16T21:16:37.530 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95822.log 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107805.log.gz 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113450.log 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102104.log.gz 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96837.log.gz 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112946.log 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95822.log.gz 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose --/var/log/ceph/ceph-client.admin.113450.log: /var/log/ceph/ceph-client.admin.102956.log 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105886.log 2023-12-16T21:16:37.531 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.113450.log.gz 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112946.log.gz 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101564.log 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105886.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.102956.log.gz --verbose 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.107469.log 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101564.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105886.log.gz 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97177.log 2023-12-16T21:16:37.532 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101564.log.gz 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111066.log 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103338.log 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107469.log.gz/var/log/ceph/ceph-client.admin.97177.log: 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110062.log 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97177.log.gz 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111066.log.gz 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95780.log 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97460.log 2023-12-16T21:16:37.533 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110062.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103338.log.gz 2023-12-16T21:16:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95780.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.110062.log.gz -- 2023-12-16T21:16:37.534 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.99689.log 2023-12-16T21:16:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97460.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95780.log.gz 2023-12-16T21:16:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97180.log 2023-12-16T21:16:37.534 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97460.log.gz 2023-12-16T21:16:37.534 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97964.log 2023-12-16T21:16:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97180.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99689.log.gz 2023-12-16T21:16:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101522.log 2023-12-16T21:16:37.535 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97180.log.gz 2023-12-16T21:16:37.535 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110798.log 2023-12-16T21:16:37.535 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97964.log.gz 2023-12-16T21:16:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108859.log 2023-12-16T21:16:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-16T21:16:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110798.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101522.log.gz 2023-12-16T21:16:37.536 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104864.log 2023-12-16T21:16:37.536 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110798.log.gz 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108859.log.gz 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104554.log 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.104864.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.97880.log 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104554.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104864.log.gz 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104554.log.gz 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.101894.log 2023-12-16T21:16:37.537 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99605.log 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97880.log.gz 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100598.log 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108687.log 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101894.log.gz 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111228.log 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99605.log.gz 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107052.log 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100598.log.gz 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112330.log 2023-12-16T21:16:37.538 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111228.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.108687.log.gz 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107721.log 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111228.log.gz 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107052.log: gzip/var/log/ceph/ceph-client.admin.112330.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.99437.log 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.107052.log.gz/var/log/ceph/ceph-client.admin.107721.log: 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112330.log.gz 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107721.log.gz 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.113618.log 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99437.log: gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99437.log.gz 2023-12-16T21:16:37.539 INFO:teuthology.orchestra.run.smithi046.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.100886.log 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103516.log 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.113618.log.gz 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103388.log 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.100886.log.gz 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101054.log 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112120.log 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103388.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.103516.log.gz 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96502.log 2023-12-16T21:16:37.540 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.103388.log.gz 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101054.log.gz 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112120.log: /var/log/ceph/ceph-client.admin.96502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96834.log 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112120.log.gz 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96502.log.gz 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112036.log 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107595.log 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108731.log 2023-12-16T21:16:37.541 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96834.log.gz 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113760.log 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107595.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112036.log.gz 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/ceph-client.admin.108731.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107595.log.gz 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.97712.log 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113760.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108731.log.gz 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113760.log.gz 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.103642.log 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100556.log 2023-12-16T21:16:37.542 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97712.log.gz 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.103642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99353.log 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100556.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.103642.log.gz 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106150.log 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100556.log.gz 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104638.log 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99353.log: /var/log/ceph/ceph-client.admin.106150.log: gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99353.log.gz 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.0.104135.log 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104638.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106150.log.gz 2023-12-16T21:16:37.543 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106658.log 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104638.log.gz 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.104135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104680.log 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101138.log 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106658.log.gz/var/log/ceph/ceph-client.admin.104680.log: 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104680.log.gz 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.96326.log 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109562.log 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101138.log.gz 2023-12-16T21:16:37.544 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96326.log: gzip -5 91.2% -- replaced with /var/log/ceph/ceph-client.0.104135.log.gz 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.100456.log 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96326.log.gz 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.0.109562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101684.log 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100456.log: gzip -5 --verbose 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100456.log.gz 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.109237.log 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101684.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101684.log.gz 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106700.log 2023-12-16T21:16:37.545 INFO:teuthology.orchestra.run.smithi046.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.109562.log.gz 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose 91.2% -- replaced with /var/log/ceph/ceph-client.admin.109237.log.gz 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.111150.log 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102524.log 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106700.log.gz 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr: 85.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95566.log 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111150.log.gz 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106868.log 2023-12-16T21:16:37.546 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.95566.log: -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.102524.log.gz 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-client.admin.102020.log 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.106868.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95566.log.gz 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106868.log.gz 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr: --verbose -- /var/log/ceph/ceph-client.admin.97628.log 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102020.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97628.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102020.log.gz 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113802.log 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97628.log.gz 2023-12-16T21:16:37.547 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96074.log 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/ceph-client.admin.113802.log: --verbose -- /var/log/ceph/ceph-client.admin.101254.log 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96074.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113802.log.gz 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99563.log 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96074.log.gz 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109069.log 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.99563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107679.log 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101254.log.gz 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.109069.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99563.log.gz 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109069.log.gz 2023-12-16T21:16:37.548 INFO:teuthology.orchestra.run.smithi046.stderr: -- /var/log/ceph/ceph-mds.c.log 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107679.log: gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107679.log.gz 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.111024.log 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112288.log 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.111024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97173.log 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.111024.log.gz 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112288.log: 92.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108339.log 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112288.log.gz 2023-12-16T21:16:37.549 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97178.log 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108339.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97173.log.gz 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107553.log 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.108339.log.gz/var/log/ceph/ceph-client.admin.97178.log: 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109836.log 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97178.log.gz 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108160.log 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107553.log.gz 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.109836.log: /var/log/ceph/ceph-client.admin.113282.log 2023-12-16T21:16:37.550 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108160.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109836.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105928.log 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108160.log.gz 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104596.log 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105928.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113282.log.gz 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108034.log 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113492.log 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105928.log.gz/var/log/ceph/ceph-client.admin.104596.log: 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.108034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101096.log 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104596.log.gz 2023-12-16T21:16:37.551 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108034.log.gz 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.113492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102912.log 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.113492.log.gz 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95354.log 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.101096.log: /var/log/ceph/ceph-client.admin.102912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108985.log 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95354.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101096.log.gz 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102912.log.gz 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95354.log.gz 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96505.log 2023-12-16T21:16:37.552 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.108985.log: /var/log/ceph/ceph-client.admin.102188.log 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96505.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110882.log 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108985.log.gz 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96505.log.gz 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100151.log 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.110882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104292.log 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.102188.log.gz 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.100151.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.110882.log.gz 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102828.log 2023-12-16T21:16:37.553 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103472.log 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100151.log.gz 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.102828.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.104292.log.gz 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96032.log 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102828.log.gz 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5/var/log/ceph/ceph-client.admin.103472.log: --verbose -- /var/log/ceph/ceph-client.admin.104948.log 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.96032.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.103472.log.gz 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102356.log 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96032.log.gz 2023-12-16T21:16:37.554 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100067.log 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.104948.log: /var/log/ceph/ceph-client.admin.102356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112862.log 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr: 88.0%/var/log/ceph/ceph-client.admin.100067.log: -- replaced with /var/log/ceph/ceph-client.admin.104948.log.gz 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102356.log.gz 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100067.log.gz 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105110.log 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112862.log: gzip -5 --verbose -- 87.5% -- replaced with /var/log/ceph/ceph-client.admin.112862.log.gz 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr: /var/log/ceph/ceph-client.admin.107214.log 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97172.log 2023-12-16T21:16:37.555 INFO:teuthology.orchestra.run.smithi046.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.105110.log.gz 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.107214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96841.log 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107214.log.gz 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112666.log 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97172.log: /var/log/ceph/ceph-client.admin.96841.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96841.log.gz 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97172.log.gz 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95608.log 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112414.log 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.95608.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112666.log.gz 2023-12-16T21:16:37.556 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95608.log.gz 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105844.log 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.112414.log: /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90203.log 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.105844.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112414.log.gz 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.105844.log.gz 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103208.log 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97544.log 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.90203.log: /var/log/ceph/ceph-client.admin.103208.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.90203.log.gz 2023-12-16T21:16:37.557 INFO:teuthology.orchestra.run.smithi046.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103208.log.gz 2023-12-16T21:16:37.558 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.97544.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97544.log.gz 2023-12-16T21:16:37.558 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-12-16T21:16:37.559 INFO:teuthology.orchestra.run.smithi046.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-16T21:16:37.559 INFO:teuthology.orchestra.run.smithi046.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-16T21:16:37.559 INFO:teuthology.orchestra.run.smithi046.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-16T21:16:37.560 INFO:teuthology.orchestra.run.smithi121.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-16T21:16:37.560 INFO:teuthology.orchestra.run.smithi121.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-16T21:16:37.560 INFO:teuthology.orchestra.run.smithi121.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-16T21:16:37.560 INFO:teuthology.orchestra.run.smithi121.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-16T21:16:37.561 INFO:teuthology.orchestra.run.smithi121.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-16T21:16:37.561 INFO:teuthology.orchestra.run.smithi121.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-16T21:16:37.561 INFO:teuthology.orchestra.run.smithi121.stderr: 2023-12-16T21:16:37.561 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m0.223s 2023-12-16T21:16:37.561 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m1.126s 2023-12-16T21:16:37.561 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.067s 2023-12-16T21:16:37.581 INFO:teuthology.orchestra.run.smithi046.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-16T21:16:37.585 INFO:teuthology.orchestra.run.smithi046.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-16T21:16:37.587 INFO:teuthology.orchestra.run.smithi046.stderr: 2023-12-16T21:16:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m0.316s 2023-12-16T21:16:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m1.207s 2023-12-16T21:16:37.587 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.249s 2023-12-16T21:16:37.588 INFO:tasks.ceph:Archiving logs... 2023-12-16T21:16:37.588 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/remote/smithi046/log 2023-12-16T21:16:37.589 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-16T21:16:38.084 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/remote/smithi121/log 2023-12-16T21:16:38.085 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-16T21:16:38.262 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-16T21:16:38.284 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-16T21:16:38.285 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-16T21:16:38.288 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-16T21:16:38.371 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-16T21:16:38.382 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-16T21:16:38.382 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:16:38.385 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:16:38.397 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:16:38.399 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:16:38.402 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:16:38.402 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2023-12-16T21:16:38.402 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -158us[ -158us] +/- 78ms 2023-12-16T21:16:38.402 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 +119us[ +119us] +/- 69ms 2023-12-16T21:16:38.402 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +36us[ +36us] +/- 77ms 2023-12-16T21:16:38.402 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:16:38.404 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:16:38.404 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-12-16T21:16:38.404 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 -148us[ -142us] +/- 78ms 2023-12-16T21:16:38.404 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 +113us[ +119us] +/- 69ms 2023-12-16T21:16:38.404 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +43us[ +49us] +/- 77ms 2023-12-16T21:16:38.404 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:16:38.405 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-16T21:16:38.413 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-16T21:16:38.414 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-16T21:16:38.443 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-16T21:16:38.560 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-16T21:16:38.657 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T21:16:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:16:38.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T21:16:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:16:38.726 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-16T21:16:38.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-16T21:16:38.860 INFO:teuthology.task.internal:Duration was 709.347343 seconds 2023-12-16T21:16:38.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-16T21:16:38.925 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-16T21:16:38.925 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T21:16:38.927 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T21:16:38.972 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:16:38.977 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:16:39.333 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-16T21:16:39.333 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2023-12-16T21:16:39.333 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-16T21:16:39.354 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2023-12-16T21:16:39.355 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-16T21:16:39.389 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-16T21:16:39.390 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-16T21:16:39.397 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-16T21:16:39.457 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-16T21:16:39.458 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-16T21:16:39.470 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-16T21:16:39.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-16T21:16:39.609 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-16T21:16:39.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T21:16:39.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T21:16:39.669 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-16T21:16:39.684 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T21:16:39.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T21:16:39.712 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2023-12-16T21:16:39.735 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2023-12-16T21:16:39.751 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-16T21:16:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:16:39.786 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-16T21:16:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:16:39.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-16T21:16:39.817 INFO:teuthology.task.internal:Transferring archived files... 2023-12-16T21:16:39.818 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/remote/smithi046 2023-12-16T21:16:39.818 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T21:16:39.869 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833/remote/smithi121 2023-12-16T21:16:39.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T21:16:39.904 INFO:teuthology.task.internal:Removing archive directory... 2023-12-16T21:16:39.905 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-16T21:16:39.907 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-16T21:16:39.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-16T21:16:39.967 INFO:teuthology.task.internal:Not uploading archives. 2023-12-16T21:16:39.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-16T21:16:39.994 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-16T21:16:39.994 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T21:16:39.997 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T21:16:40.009 INFO:teuthology.orchestra.run.smithi046.stdout: 265505 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 21:16 /home/ubuntu/cephtest 2023-12-16T21:16:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 21:16 /home/ubuntu/cephtest 2023-12-16T21:16:40.012 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-16T21:16:40.028 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-16T21:16:40.094 INFO:teuthology.nuke:Checking targets against current locks 2023-12-16T21:16:40.122 DEBUG:teuthology.nuke:shortname: smithi046 2023-12-16T21:16:40.123 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:16:40.147 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:56:13.427323', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1a5SboxJoSrpG9mBywr6CCPu5P8Z3x4NaoXLuZ0bf4KeJcCsXygvslEt6eyjzJZfTB9xRERW+QgLRGV0099aw='} 2023-12-16T21:16:40.151 DEBUG:teuthology.nuke:shortname: smithi121 2023-12-16T21:16:40.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:16:40.176 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494833', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 20:56:13.425727', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCbkN8s8lHKeTZqUwEuj1Kn16eqSfxnbUrx/iZfNDoz3x1UwK7+tQOL2YiAs2mO3QT6yiHfQCZ6uKmanhTc6OTk='} 2023-12-16T21:16:40.204 INFO:teuthology.orchestra.console.smithi046:Power off 2023-12-16T21:16:40.204 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T21:16:40.229 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2023-12-16T21:16:40.230 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:16:40.247 INFO:teuthology.orchestra.console.smithi121:Power off 2023-12-16T21:16:40.248 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T21:16:40.258 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2023-12-16T21:16:40.270 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2023-12-16T21:16:40.271 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:16:40.294 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-12-16T21:16:44.260 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:16:44.296 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:16:44.371 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2023-12-16T21:16:44.406 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2023-12-16T21:16:48.373 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:16:48.408 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T21:16:48.485 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2023-12-16T21:16:48.518 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2023-12-16T21:16:48.587 INFO:teuthology.orchestra.console.smithi046:Power off completed 2023-12-16T21:16:48.619 INFO:teuthology.orchestra.console.smithi121:Power off completed 2023-12-16T21:16:48.724 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} duration: 709.3473429679871 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1b5283e0716040b6ab0585e33cbafc9c status: fail success: false 2023-12-16T21:16:48.725 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T21:16:48.818 INFO:teuthology.run:FAIL